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A.  INTRODUCTION 

We  hypothesized  that  our  fusion  of  genetic,  visual,  and  tactile  information  would 
improve  surgeons’  understanding  of  the  extent  of  disease  and  will  ultimately  permit 
surgeons  to  better  plan  operations  and  to  prepare  for  the  actual  pathology  found.  We 
proposed  to  combine  current  medical  imaging  technologies  with  genetic  imaging  to 
leverage  our  ability  to  present  and  use  multi-sensory  imaging  information.  We  planned 
not  only  to  view,  but  also  to  physically  interact  with  a  virtual  construct  of  a  specific 
patient’s  tissues  and  organs  in  a  manner  that  would  be  immediately  intuitive  and  useful 
to  a  surgeon.  We  hypothesized  that  in  silico  practice  would  shorten  the  time  of 
surgeries  and  reduce  surgical  complications.  For  genetic  imaging  of  pancreatic  cancer 
gene  expression,  we  proposed  to  design  a  new  ligand  for  EGFR  by  molecular  dynamics, 
with  haptic  sensing  of  the  kinetic  binding  pathway.  For  in  silico  practice  surgery  on  an 
individual  patient,  we  proposed  merging  of  pancreatic  cancer  PET/CT  scans  into  3D 
images,  and  haptic  manipulation  of  PET/CT  pancreatic  cancer  images. 


B.  BODY 


Task  1.1.  Acquisition  of  molecular  dynamics  hardware  and  software  (Leads: 
WickstromThakur) 

Milestone  1.1.1.  Acquisition  and  installation  of  computer  plus  Amber  10  molecular 
dynamics  software  packages.  Quotes  will  be  obtained,  permission  from  TATRC  will  be 
requested,  purchase  orders  will  be  issued,  computer  hardware  and  software  will  be 
received,  installed,  and  put  into  operation. 


•  Amber  10  [4]  acquired  and  installed  on  Linux  64-bit  High  Performance  Computer 
Cluster  “Egypt”  at  TJU. 


VH  Dissector  and  3D  Slicer  [5]  acquired  and  installed  on  existing  Dell  Inspiron 


531. 

The  volumetric-haptic  3DBoxx  8570 
computer  system  “Kollman”  was 
delivered  from  Boxx  Technologies  on 
12  Feb  2010.  Through  negotiations, 
we  spent  about  half  as  much  as 
originally  budgeted  for  the  3D  haptic 
system. 

3D  projections  of  peptides  (Fig.  1), 
PNAs,  and  RNAs  on  Kollman  with 
PyMol  were  sharp  and  clear  from 
across  Room  209  when  wearing  the 
specified  white  light-controlled  3D 
glasses. 


Fig.  1.  3D  projection  of  EGF20-31,  after 
energy  minimization,  on  the  3D  Boxx 
8570  Kollman  at  TIU. 
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•  3D  Slicer  [5],  Amber  11  [4],  and  Amira  5  (http ://www. amira. com/amira.html) 
acquired  and  installed  on  Kollman  and  on  Egypt. 

Task  1.2.  Molecular  design  of  EGFR  ligand  for  gene  imaging  probe  (Leads: 
Wickstrom/Thakur) 

Milestone  1.2.1.  Selection  of  ligand  for  EGFR  uptake  of  gene  imaging  probe.  In  view  of  the 
widespread  overexpression  of  EGFR  on  cancer  cells,  generating  a  successful  endocytic 
analog  of  EGF  for  mounting  on  our  mRNA  probes  could  expand  the  spectrum  of  cancers 
for  imaging.  EGFR,  (ErbB-1,  Her-1),  is  a  1250-aa,  165  kDa  transmembrane  glycoprotein 
tyrosine  kinase  cell  surface  protein  that  binds  EGF,  a  53-aa,  6  kDa  extracellular  signaling 
peptide.  Based  on  the  existing  crystal  structure  of  EGF-EGFR,  we  will  select  a  peptide  loop 
of  EGF  that  interacts  with  EGFR  as 
our  first  candidate  ligand. 

•  EGF  central  loop,  amino 
acids  20-31, 

CMYIEALDKYAC ,  with  Cys- 
Cys  disulfide  bridge 
between  Cys20  and  Cys31 
(Fig.  2B),  was  selected  in 
Q1  from  the  crystal 
structure  [1]  as  the  first 
candidate  ligand.  The  linear 
version  (Fig.  2A)  is 
expected  to  serve  as  a 
control. 

•  EGF  amino  acids  32-48, 

NCWGYIGERCQYRDLK 
(Fig.  2C),  was  selected  in 
Q2  as  a  second  candidate 
ligand,  based  on  binding  to 
EGFR  on  cells  [2],  Cys-Cys 
cyclized  version  (Fig.  2D) 
also  exhibited  EGFR 
binding. 

•  GE 1 1 ,  YHWYGYTPQNVI,  an 
EGFR-binding  peptide 
identified  by  phage  display 
[3]  (Fig.  2E),  was  selected 
as  a  third  candidate  ligand, 
along  with  its  Cys-Cys 
cyclized  version  (Fig.  2F). 


Fig.  2.  Potential  EGF  ligands,  energy-minimized  by 
molecular  dynamics  in  Chem3D.  A,  EGF20-31 
selected  from  crystal  structure  [1];  B,  cyclized  EGF20- 
31;  C,  EGFR-binding  EGF32-48  identified  by  cell 
binding  [2];  D,  latter  peptide  cyclized  via  Cys33  and 
Cys42;  E,  EGFR-binding  peptide  identified  by  phage 
display  [3];  F,  latter  peptide  cyclized  by  N  and  C 
terminal  cysteines. 
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E  (kcal/mol) 


Milestone  1.2.2.  Molecular  dynamics  calculations  of  selected  EGFR  ligand  binding 
(1000  ps  minimum  energy).  Calculation  of  the  most  probable  structure  of  the 
selected  EGFR  ligand  in  water  will  predict  its  ability  to  bind  to  its  intended  EGFR 
target.  Using  Amber  10  in  house,  or  on  Salk  at  the  Pittsburgh  Supercomputer 
Center,  we  will  calculate  the  potential  function,  U(x),  and  solve  the  negative 
gradient  of  the  potential 
energy  function,  -VU(x), 
along  the  kinetic  docking 
pathways  for  the  selected 
peptide  loop  of  EGF  that 
interacts  with  EGFR,  at  every 
1 A  of  separation  between 
the  peptide  and  EGFR  from 
30  A  out  to  0  A,  the  final 
docked  state.  Using  the  two 
manipulators,  we  will  guide 
the  candidate  peptide  into 
the  EGFR  binding  site,  from 
30  A  separation  to  0  A,  with 
haptic  responses  controlled 
by  the  docking  calculations. 

We  will  sense  kinetic 
barriers,  or  difficult 
rearrangements,  through  our 
hands,  as  well  as  by 
examining  the  numbers  from 
the  calculation.  The  criterion 


— -  1  (PS) 

vs 

dk>rte  30 

- - Ups) 

vs 

EGF  i'INOLj 

-  Ups) 

vs 

EGF  20-31 

Ups) 

vs 

EGF  32*43 

- - Ups) 

vs 

EGF  33-42 

Fig.  3.  Total  energy  from  MD  simulations  of  EGF 
peptides  in  water,  calculated  with  Amber  10  over 
9  ns.  EGF32-48  showed  greater  stability  than 
EGF20-31. 


for  favorable  energetics  of 
binding  to  EGFR  will  be  a 
predicted  Kd  <  100  nM. 
Unfavorable  energetics 
would  be  indicated  by  a 
predicted  Kd  >  1  /xM. 
Molecular  dynamics 
simulations  of  EGF  and  EGF 
peptides  over  9  ns  were 
performed  for  the  fragments 
chosen  in  Fig.  2,  in  the 
presence  of  explicit  water 
molecules.  Large  scale  Amber 
10  calculations  required  use 
of  Salk  at  the  Pittsburgh 
supercomputer  system.  Total 
energy  plots  were 
constructed  to  evaluate 
stability  of  each  peptide  in 


Fig.  4.  sEGFR  intramolecular  and  intermolecular 
interactions  that  are  involved  in  receptor 
autoinhibition  and  EGF  binding.  The  dimerization 
arm  of  domain  II  is  capable  of  forming  several 
noncovalent  interactions  with  domain  IV  to  prevent 
formation  of  stable  dimers  and  restrict 
conformational  sampling  of  sEGFR. 
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explicit  solvent.  EGF  32-48  had 
the  lowest  peptide  fragment 
energy  after  equilibration  and 
the  high  affinity  mutant  EGF  had 
a  profile  similar  to  that  of  wild 
type  EGF  (Fig.  3).  The  increase 
in  energy  most  likely  results 
from  increased  entropy  due  to 
size  and  amino  acid  composition 
of  each  cyclic  peptide.  EGF32-48 
has  two  tryptophans  near  the  C 
terminus  that  may  participate  in 
base  stacking,  decreasing 
entropy  and  stabilizing  its 
structure. 

•  Minimum  energies  of  the 
extracellular  ligand  binding 
domain  of  EGFR,  with  and 
without  EGF  bound,  were  calculated  in  the  presence  of  water  with  Amber  11  [6] 
(Appendix).  We  demonstrated  that  EGF  binding  to  the  extracellular  domain  of 
EGFR  (sEGFR)  disrupts  auto-inhibitory  intramolecular  interactions  (Fig.  4)  by 
increasing  the  distance  between  the  so-called  “dimerization  arm”  and  domain  IV 
(Fig.  5).  To  investigate  early  ligand  binding  effects  on  sEGFR  dimersfwe 
constructed  a  singly  ligated  sEGFR  dimer  for  simulation.  MD  simulations  of  the 
singly  ligated  sEGFR  dimer  exhibited  asymmetrical  motion  in  each  monomer  and  a 
decrease  in  the  size  of  the  second  EGF  binding  pocket  by  10  A  (Fig.  6).  Free  energy 
calculations  indicated  that  asymmetrical  dimerization  with  only  one  EGF-sEGFR 
monomer  is  more  energetically  favorable  than  dimerization  of  two  ligated  sEGFR 
monomers  by  2.88  kcal/mol. 

•  The  molecular  dynamics  docking 
pathway  of  EGF  into  the  ligand  binding 
pocket  of  EGFR  was  then  calculated 
with  ZDOCK,  an  online  server  that 
performs  rigid  body  docking 
calculations  and  determines  their 
relative  energies  in  implicit  water  with 
the  appropriate  dielectric  constant,  on 
Salk  at  the  Pittsburgh  supercomputer 
system  (Fig.  7).  We  discovered  that  the 
entire  kinetic  docking  pathway  could 
be  calculated  in  a  single  run,  beginning 
with  EGF  outside  of  the  EGFR  ligand 
binding  site.  The  ZDOCK  force  fields 
were  sufficient  to  draw  EGF  into  the 
ligand  binding  pocket.  This  insight 
eliminates  the  need  for  separate 


Fig.  6.  MD  simulation  predicts  that  EGF 
binding  forces  the  dimerization  arm  to 
disengage  from  domain  IV.  Separation 
distance  between  the  dimerization  arm 
and  the  S558-S567  loop  of  the  tethering 
arm  of  domain  IV  increased  upon  EGF 
binding. 


Fig.  5.  The  binding  pocket  of  sEGFR  coordinates 
EGF  (blue)  with  both  domains  I  and  III.  Several 
residues  are  responsible  for  ligand  coordination. 
Their  distances  are  symmetric  in  the  sEGFR 
dimer  crystal  structure. 
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calculations  over  a  series  of  specific 
separations. 

•  MD  simulations  for  EGFR 
complexes  with  the  candidate 
peptides  of  Fig.  2  were  also 
performed  and  energy  plots  were 
generated  (Fig.  8).  The  EGFR 
receptor  alone  (EGFR  apo) 
displayed  the  lowest  energy,  while 
the  two  complexes  showed  slightly 
increased  energies  overall.  For  EGF 
binding  to  EGFR,  AG  =  -68±10 
kcal/mol.  For  EGF32-48  binding  to 
EGFR,  AG  =  -40±8  kcal/mol.  This 
could  be  a  result  of  conformational 
changes  occurring  and  interactions 
between  the  peptides  and  the 
receptor.  Neither  complex 
underwent  substantial 
conformational  changes  within  this 
time  frame,  suggesting  that  longer 
simulation  times  need  to  be 
performed  in  order  to  capture  the 
receptor  activation  event.  The 
molecular  dynamics  insight  above 
that  EGFR  activation  requires  a 
conformational  shift  that  depends  on 
all  domains  of  EGF  implies  that  a 
single  loop  of  EGF  will  be  insufficient 
to  induce  internalization  of  EGFR 
with  cargo. 


Task  1.3.  Synthesis  and  validation  of 
EGFR  ligand  for  gene  imaging  probe 
(Leads:  Wickstrom/Thakur) 

Milestone  1.3.1.  Synthesis  and  purification 
of  chelator-PNA  gene  probe  with  selected 
EGFR  ligand  (95%  homogeneity).  The 
energetically  optimal  peptide  selected  above 
will  be  extended  from  a  solid  phase  support 
by  solid  phase  coupling  of  Fmoc-l-amino 
acid  monomers.  The  terminal  Cys  residues 
will  be  cyclized.  A  diethyleneglycol  spacer 


Fig.  7.  Graphic  image  of  EGF  docking  with 
EGFR  at  20  A  out  of  the  ligand  binding 
pocket,  during  a  kinetic  docking  pathway 
molecular  dynamics  energy  minimization 
using  ZDOCK  on  the  Salk  64-bit  high 
performance  computer  of  the  Pittsburgh 
supercomputer  system.  This  structure  was 

aft pr  9SO  ns  of  motion 
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Fig.  8.  Total  energy  from  MD  simulations 
of  EGF  and  EGF32-48  interacting  with 
EGFR  in  water,  calculated  with  Amber  10 
over  4-10  ns. 
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will  be  coupled,  followed  by  the  KRAS2  G12D  PNA  sequence,  a  second  diethyleneglycol 
spacer,  then  a  DOTA  chelator.  The  finished  D03A-PNA-peptide  will  be  cleaved  from  the 
solid  support,  deprotected,  purified  by  reversed  phase  liquid  chromatography,  and 
analyzed  by  mass  spectroscopy  to  determine  whether  the  desired  sequence  was 
synthesized  correctly. 

•  AlexaFluor  532-EGF20-31  (Fig.  9)  was  synthesized  by  solid  phase  coupling.  F1PLC 
and  MS  (Fig.  10)  were  satisfactory. 


0.00  2.00  41  611  8.00  10*  Ilf  141  161  181  211  111  241  261  281  301 
Hute$ 


m  /z 


Fig.  10.  Characterization  of  AlexaFluor  532-EGF20-31.  Left:  preparative  reversed  phase 
liquid  chromatography  of  crude  cleavage  product  separated  by  a  30  min  gradient  from 
10%  to  60%  CH3CN  in  aqueous  0.1%  CF3C02H,  at  1  mL/min,  on  4.5  x  250  mm  C18 
column,  at  50°C,  detected  at  260  nm.  The  main  peak  is  the  desired  peptide.  Right: 
MALDI-TOF  mass  spectroscopy  after  chromatographic  purification.  The  purified  main 
peak  was  excited  from  an  a-cyano-hydroxycinnamic  acid  matrix,  yielding  a  mass  of 
2459.83  Da  (M+H)  (calculated:  2458.87  Da). 
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•  AlexaFluor  532-EGF32-48  (Fig.  11)  was  synthesized  by  solid  phase  coupling. 
HPLC  and  MS  (Fig.  12)  were  satisfactory. 


Fig.  12.  Characterization  of  AlexaFluor  532-EGF32-48.  Left:  preparative  reversed  phase 
liquid  chromatography  of  crude  cleavage  product  separated  by  a  40  min  gradient  from 
15%  to  65%  CH3CN  in  aqueous  0.1%  CF3C02H,  at  3  mL/min,  on  4.5  x  250  mm  C18 
column,  at  rt,  detected  at  260  nm.  The  main  peak  is  the  desired  peptide.  Right:  MALDI- 
TOF  mass  spectroscopy  after  chromatographic  purification.  The  purified  main  peak  was 
excited  from  an  a-cyano-hydroxycinnamic  acid  matrix,  showing  the  predicted  mass  of 
3036.5  Da  (M+H). 
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•  AlexaFluor  532-GE11  (Fig.  13)  was  synthesized  by  solid  phase  coupling.  HPLC 
and  MS  (Fig.  14)  were  satisfactory. 


Fig.  14.  Characterization  of  AlexaFluor  532-GE11.  Left:  analytical  reversed  phase  liquid 
chromatography  of  purified  agent  by  a  30  min  gradient  from  20%  to  75%  CH3CN  in  aqueous 
0.1%  CF3C02H,  at  1  mL/min,  on  4.5  x  250  mm  C18  column,  at  rt,  detected  at  260  nm.  Right: 
MALDI-TOF  mass  spectroscopy  after  chromatographic  purification.  The  purified  main  peak 
was  excited  from  an  a-cyano-hydroxycinnamic  acid  matrix,  showing  the  predicted  mass  of 
2563. 98Da  (M+H). 
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•  Alexa  Fluor  532-KRAS2  PNA-GE11  (Fig.  15)  was  synthesized  by  solid  phase 
coupling.  FIPLC  and  MS  (Fig.  16)  were  satisfactory. 


m/z 


Fig.  16.  Characterization  of  AlexaFluor  532-KRAS2  PNA-GE11.  Left:  preparative 
reversed  phase  liquid  chromatography  of  crude  cleavage  product  separated  by  a  40 
min  gradient  from  10%  to  55%  CH3CN  in  aqueous  0.1%  CF3C02H,  at  3  mL/min,  on  4.5 
x  250  mm  C18  column,  at  50°C,  detected  at  260  nm.  The  main  peak  is  the  desired 
KRAS2  agent.  Right:  MALDI-TOF  mass  spectroscopy  after  chromatographic 
purification.  The  purified  main  peak  was  excited  from  an  a-cyano-hydroxycinnamic 
acid  matrix,  showing  the  predicted  mass  of  5883.2  Da  (M+2H). 
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DOTA-KRAS2  PNA-EGF32-48  with  cyclized  peptide  ring  (Fig.  17)  was  synthesized  by 
solid  phase  coupling.  HPLC  and  MS  (Fig.  18)  were  satisfactory. 
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Fig.  17.  Structure  of  DOTA-AKAS2  PNA-EGF32-48  with  1  Na+,  2  K+. 


m  w 


ms 


Fig.  18.  Characterization  of  DOTA -KRAS2  PNA-EGF32-48,  WT5863.  Left:  preparative 
reversed  phase  liquid  chromatography  of  crude  cleavage  product  separated  by  a  30  min 
gradient  from  15%  to  65%  CH3CN  in  aqueous  0.1%  CF3C02H,  at  3  mL/min,  on  4.5  x  250 
mm  C18  column,  at  50°C,  detected  at  260  nm.  The  main  peak  is  the  desired  KRAS2  agent. 
Right:  MALDI-TOF  mass  spectroscopy  after  chromatographic  purification.  The  purified 
main  peak  was  excited  from  an  a-cyano-hydroxycinnamic  acid  matrix,  showing  the 
predicted  mass  of  5863. 83  Da  (M+Na+2K). 
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Milestone  1.3.2.  Validation  of  chelator-PNA  gene  probe  with  selected  EGFR  ligand  by 
binding  to  cells  expressing  EGFR  vs.  control  peptide  (agreement  with  calculation,  5-fold 
greater).  The  purified  D03A-PNA-peptide  will  be  labeled  with  64Cu,  then  used  to  measure 
the  binding  affinities  of  the  [64Cu]chelator-PNA-peptides  to  EGFR  on  human  AsPCl 
pancreas  cancer  cells  that  overexpress  EGFR,  plus  blocking  controls  with  full-length 
unlabeled  EGF  to  test  for  EGFR  specificity. 

•  We  measured  by  confocal  microscopy  the  uptake  of  AlexaFluor488-EGF, 

AlexaFluor488-EGF3248,  and  AlexaFluor647-EGF2031  by  AsPCl  pancreas  cancer 
cells  that  overexpress  EGFR.  Fluorescent  agents  were  synthesized  to  enable 
visualization  of  intracellular  localization.  AsPCl  cells  were  seeded  in  8-chamber 
slides  (Lab-Tek,  Nunc  Inc.)  and  grew  to  50%  confluency.  Cells  in  each  chamber 
were  washed  once  and  incubated  for  1  hr  at  37°C  in  EGF-AlexaFluor488  (50  nM) 
or  EGF3248-AlexaFluor488  (50  nM  or  1  /xM)  in  complete  RPMI  1640  medium.  At 
the  end  of  incubation,  cells  were  washed  three  times  with  PBS  and  fixed  with  1% 
paraformaldehyde  in  PBS.  Then,  the  chamber  structures  were  removed  and 
slides  were  covered  with  coverslips  for  imaging.  Confocal  images  were  taken  on  a 
Zeiss  LSM  510  Meta  confocal  microscope  with  a  40x/1.3  oil  DIC  objective.  At  50 
nM,  internalized  EGF-AlexaFluor488  molecules  were  concentrated  as  puncta 
throughout  the  cytoplasm  of  AsPCl  cells,  but  not  in  the  nucleus.  The 
characteristic  could  be  the  result  of  concentrated  molecules  in  vesicles  (Fig.  19). 


Fig.  19.  Internalization  of  50  nM  AlexaFluor488-EGF  binding  to  AsPCl  cells  for  1  hr 
at  37°C,  recorded  on  a  Zeiss  LSM  510  Meta  confocal  microscope  with  a  40x/1.3  Oil 
DIC  objective.  Excitation:  488  nm.  Emission:  505-550  nm.  Righthand  image  shows 
2x  zoom  of  center  of  left  image. 
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•  After  one  hr  of  incubation  at  37°C,  AlexaFluor488-EGF3248  showed  moderate 
internalization  (Fig.  20).  With  50  nM  AlexaFluor488-EGF3248,  the  fluorescent 
signals  are  distributed  outside  the  nucleus.  With  1  /xM  AlexaFluor488-EGF3248, 
the  fluorescent  signals  are  distributed  throughout  the  entire  cell.  This  could  be 
explained  by  different  mechanisms  of  internalization.  At  low  concentration  of 
peptide,  the  internalization  mechanism  could  be  clathrin-mediated  endocytosis 
At  high  concentration  of  peptide,  the  internalization  mechanism  could  also  be 
pinocytosis. 


•  In  contrast,  even  1  /xM  AlexaFluor647-EGF2031  showed  no  uptake  by  AsPCl 
pancreas  cancer  cells  after  1  hour  at  37°C  (Fig.  21).  Furthermore,  AlexaFluor532- 
GE11  was  taken  up  equally  well  by  AsPCl  cells,  that  overexpress  EGFR,  and  by 
CHO  cells,  that  do  not  make  human  EGFR  (not  shown),  implying  that  GE11 
uptake  is  nonspecific  to  EGFR. 


c  *  C( 


\  « 

5 

C  /  c  s  c 
t  K  cr 


Fig.  21.  Internalization  of  AlexaFluor647-EGF2031  binding  to  AsPCl  cells 
for  5  hr  at  37°C,  recorded  on  a  Zeiss  510  Meta  confocal  microscope  with  a 
40x/1.3  oil  DIC  objective.  Excitation:  532  nm.  Emission:  554  nm.  Left: 
fluorescent  image.  Right:  bright  field. 
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•  We  measured  by  flow 
cytometry  the  uptake  of 
AlexaFluor488-EGF  (Fig.  22), 
AlexaFluor488-EGF20-31  (Fig. 
23)  and  AlexaFluor488- 
EGF3248  (Fig.  23)  byAsPCl 
pancreas  cancer  cells  that 
overexpress  EGFR.  Cells  were 
seeded  in  10-cm  cell  culture 
dishes  and  grew  to  80% 
confluency.  Cells  were  then 
incubated  in  complete 
medium  containing  50  nM 
EGF-AlexaFluor488  or 
EGF3248-AlexaFluor488  at  4°C 
for  2  hr.  After  that,  cells  were 
washed  three  times  with  cold 
PBS,  followed  by  incubation  at 
37°C  for  30  min  to  allow 
internalization.  Non-specific 
binding  was  measured  by 
adding  a  20-fold  excess  of 
EGF.  After  that,  cells  were 
collected  by  using  a  cell 
scraper  and  suspended  in 
FACS  buffer  (2%  fetal  bovine 
serum  and  0.01%  sodium 
azide  in  PBS)  before  fixation 
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Fig.  22.  AlexaFluor488-EGF  binding  constant 
determined  by  flow  cytometry.  105  AsPCl 
cells  were  incubated  with  a  ramp  of 
AlexaFluor488-EGF  concentrations  (0, 
0.00625,  0.0625,  0.125,  0.25,  0.5,  1,  2,  4,  8, 
16,  32,  64  nM)  for  two  hr  at  4°C.  Non-specific 
binding  was  measured  by  adding  20-fold 
excess  of  EGF.  Ex:  495  nm  Em:  519  nm 
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Fig.  23.  Fluorescent  EGF20-31  and  EGF32-48  uptake  determined  by  flow  cytometry.  105 
AsPCl  cells  were  incubated  with  a  ramp  of  EGF20-31  and  EGF32-48  (0,  0.00625,  0.0625, 
0.125,  0.25,  0.5,  1,  2,  4,  8,  16,  32,  64  nM)  for  two  hr  at  4°C.  Non-specific  binding  was 
measured  by  adding  20-fold  excess  of  EGF.  Ex:  495  nm  Em:  519  nm 
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by  adding  an  equal  volume  of  1% 
paraformaldehyde.  Fluorescence 
emission  due  to  internalized 
fluorophore-EGF/EGF3248  was 
determined  by  flow  cytometry  on  a 
BD  FACS  Calibur.  The  starting 
fluorescence  intensity  of  cells  after 
fluorophore-EGF/EGF3248  binding 
but  without  excess  unlabeled  EGF 
was  set  to  100%.  AlexaFluor488-EGF 
binding  fit  a  hyperbolic  curve  with  a 
Kp  fit  to  1.24  nM,  but  AlexaFluor488- 
EGF20-31  (Fig.  23)  and 
AlexaFluor488-EGF3248  (Fig.  23)  fit 
a  linear,  nonspecific  binding  plot. 

Uptake  of  [64Cu]DOTA-KRAS2  PNA- 
EGF32-48  (WT5863)  byAsPCl  cells 
showed  nonspecific  uptake  (Fig.  24). 

106AsPC1  cells  were  incubated  in 
DMEM  growth  medium  with 
increasing  concentrations  of 
[64Cu]DOTA-KRAS2  PNA-EGF32-48  at 
37°C  for  60  min.  Cells  were  sedimented  at  450xg  for  5  min,  and  the  supernatants 
were  saved.  Cells  were  washed  twice  with  DMEM.  The  three  supernatants  were 
combined.  Cell  pellets  were  then  lysed  with  1  M  NaOH.  Cell  pellet  gamma  counts 
yielded  bound  agent  vs.  free  agent  in  supernatants. 

Both  the  fluorescence  and  radiouptake  results  imply  that  EGF32-48  induces 
insufficient  internalization  for  effective  imaging  of  KRAS2  mRNA.  This  result  agrees 
with  the  molecular  dynamics  insight  [6],  above,  that  EGFR  activation  requires  a 
conformational  shift  that  depends  on  all  domains  of  EGF,  not  just  one  loop. 


Fig.  24.  Uptake  of  [64Cu]DOTA-KRAS2  PNA- 
EGF32-48  determined  by  gamma  counting 
of  cell  pellets  vs.  supernatants.  106  AsPCl 
cells  were  incubated  with  a  ramp  of 
[64Cu]DOTA-KRAS2  PNA-EGF32-48  (0,  0.78, 
1.56,  3.13,  6.25,  12.5,  25,  50,  100  nM)  for 
two  hr  at  37°C.  Cell  pellet  counts  yielded 
bound  agent  vs.  free  agent  in  supernatants. 


Task  2.1.  Acquisition  of  haptic  hardware  and  software  (Leads:  Wickstrom/Thakur) 

Milestone  2.1.1.  Acquisition  and  installation  of  computer  plus  Amira  5  simulation  and 
Phantom  software  packages.  Quotes  will  be  obtained,  permission  from  TATRC  will  be 
requested,  purchase  orders  will  be  issued,  computer  hardware  and  software  will  be 
received,  installed,  and  put  into  operation. 

•  Omni  Manipulators  and  Phantom  software  acquired  and  installed  on  existing  Dell 
Inspiron  531  “Semmelweis”. 

•  The  volumetric-haptic  3DBoxx  8570  computer  system  “Kollman”  was  delivered 
from  Boxx  Technologies  on  12  Feb  2010.  Through  negotiations,  we  spent  about 
half  as  much  as  originally  budgeted  for  the  3D  haptic  system. 
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•  Amira  5  was  purchased  from  Visage 
Imaging,  San  Diego,  and  installed  on 
Kollman. 

•  SOFA  was  downloaded  from  the 
supporting  CNRS  lab  in  Lyon,  France, 
and  installed  on  Kollman. 

Task  2.2.  Acquisition  of  pancreatic  cancer 
FDG  PET/CT  data  (Leads:  Thakur/Kairys) 

Milestone  2.2.1.  Amira  5  integration  and 
conditioning  of  pancreatic  cancer  FDG 
PET/CT  data  to  a  resolution  of  5  mm.  PET 
imaging  scans  of  FDG  accumulation  in  suspect 
human  pancreatic  cancers  plus  anatomical  CT 
images  will  be  registered  to  a  common 
coordinate  system,  appropriately  scaled  and 
aligned  to  account  for  differences  in 
acquisition  and  patient  conditions  (e.g., 
movements,  positioning,  etc.),  and  stored  in  a 
common  (lossless)  format  for  subsequent 
visual  and  haptic  display. 


Fig.  25.  3D  projection  of  thorax  of 
patient  A  on  the  3D  Boxx  8570  at  TJU. 
Advanced  pancreatic  tumor  (yellow) 
extends  from  the  head  of  the  pancreas 
(green).  The  liver  (burgundy),  aorta 
(orange),  and  kidneys  (blue)  are  also 
emphasized. 


•  CT  data  and  PET  data  were  obtained  with  IRB  and  HRPO  approval  for  one  deceased 
TJU  pancreatic  cancer  patient,  de-identified  as  Patient  A. 

•  CT  data  at  1  mm  resolution  were  rendered  into  a  3D  image  using  Slicer  3,  prior  to 
acquisition  of  Amira  5  (Fig.  25). 

•  PET  and  CT  data  at  5  mm  resolution  were  rendered  into  a  3D  image  using  3DSlicer 
3  (Fig.  26). 


Fig.  26.  3D  projection  of  fused  PET/CT  data  at  5  mm  resolution  of  patient  A  on  the  3D  Boxx 
8570  at  TJU.  CT  images  are  shown  in  bone,  with  organs  in  gray,  while  PET  images  are 
emphasized  in  red. 
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Milestone  2.2.2.  Common  multilayer 
visualization  of  pancreatic  cancer  FDG 
PET/CT  data  in  layers  to  a  thickness  of  5 
mm.  Once  the  data  are  properly 
conditioned,  common  multilayer 
visualization  will  be  applied.  Thus 
individual  modalities  are  to  be  presented 
as  overlays  with  user  preferred 
transparency.  Joint  visual-haptic 
techniques  will  be  developed  to 
modulate  individual  modality  displays 
with  images,  slabs,  or  surfaces  of  the  CT 
data  while  simultaneously  feeling  the 
appropriately  aligned  FDG  PET  imagery. 

•  Visualizations  of  the  Patient  A 
thorax  were  carried  out  with 
Amira  5.  Version  1.2  was  based  on 
one  non-contrast  CT  at  1  mm 
resolution  and  one  PET  image  at  5 
mm  resolution  co-registered  with  the  non-contrast  CT.  The  surgeons  were  shown 
how  to  manipulate  the  image,  especially  how  to  strip  off  overlaying  tissues  and 
organs  overlaying  the  pancreas.  Snapshots  from  version  1.2  are  shown  in  Figs.  27 
and  28.  The  PET  image  of  the  malignant  region  of  the  pancreas  appears  in  Fig.  29 


Fig.  27.  Volumetric  rendering  of  Patient  A  in 
version  1.2.  Structures  displayed  in  this 


rendering  include  the  skeleton  (gray),  liver 
(brown),  intestines  (flesh),  stomach  (flesh), 
pancreas  (gold),  and  aorta  (red). 


Fig.  28.  Version  1.2  with  some  organs 
stripped  away  to  more  fully  display  the 
pancreatic  tumor  (black  arrow).  Organs 
displayed  in  this  rendering  include  the 
skeleton  (gray),  kidneys  (brown), 
duodenum  (flesh),  spleen  (plum),  adrenal 
gland  (yellow),  aorta  (red),  vena  cava  (blue), 
superior  mesenteric  artery  (dark  red),  and 
pancreas  (gold). 


Fig.  29.  3D  PET/CT  rendering  of  pancreas 
(gold)  and  advanced  pancreatic  tumor 
(gold)  of  patient  A.  FDG-accumulating  zone 
(pink)  reveals  aggressive,  malignant  cells. 
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•  Version  1.3  included  a  more  complete  vascular  network  due  to  the  inclusion  of  a 
contrast-enhanced  CT  at  1  mm 
resolution  and  one  PET  image  at 
5  mm  resolution.  Snapshots 
from  version  1.3  appear  in 
Figures  30,  31,  and  32. 

Discussions  with  the  evaluating 
surgeons  (see  Task  2.4  below) 
brought  forth  the  idea  of 
coloring  the  veins  more  in  line 
with  their  perceived  oxygenation 
levels.  Tilting  the  image  forward 
revealed  the  organs  behind  the 
pancreas  (Fig.  32).  To  the  right 
is  the  spleen  (plum).  Under  the 
spleen  is  the  left  kidney 
(brown).  At  the  top  middle  is  the 
left  adrenal  gland  (yellow).  In 
this  patient,  the  splenic  vein  was 
thrombosed,  leading  to  the 
enlarging  of  accessory  veins 
(purple)  leaving  the  spleen. 


Fig.  30.  Volumetric  rendering  of  Patient  A  in 
version  1.3.  Structures  displayed  in  this 
rendering  include  the  skeleton  (gray),  liver 
(brown),  intestines  (coral),  stomach  (flesh), 
right  gastroepiploic  vein  (purple),  and 
common  iliac  artery  (red). 


Fig.  31.  Volumetric  rendering  of  Patient  A  in 
version  1.3.  Structures  displayed  in  this 
rendering  include  the  skeleton  (gray),  kidney 
(brown),  pancreas  (gold),  aorta  (red), 
accessory  arteries  (red-orange)  vena  cava 
(blue),  right  gastrepiploic  and  splenic  veins 
(purple). 


Fig.  32.  Volumetric  rendering  of  Patient  A  in 
version  1.3,  tilted  forward:  pancreas  (gold), 
adrenal  gland  (yellow),  kidney  (brown), 


spleen  (plum),  aorta  (red),  accessory 
arteries  (red-orange)  vena  cava  (blue),  right 
gastrepiploic  and  splenic  veins  (purple). 
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Task  2.3.  3D  viewing  and  haptic  manipulation  of  pancreatic  cancer  PET/CT  images  (Leads: 
Barner/Steiner) 

Milestone  2.3.1.  Point-based  3D  animations  of  merged  pancreatic  cancer  FDG  PET/CT  data. 
Simulations  of  deformable  objects  will  be  carried  out  with  theoretically  justified  physics- 
based  approaches,  including  mesh  and  mesh-free  methods.  In  particular,  the  mesh-based 
mass  spring  and  finite  element  method  (FEM)  will  be  applied  to  create  haptic  responses  to 
tissues  seen  in  images.  The  combination  of  mesh-free  physics  with  point-sampled  surfaces, 
yielding  the  so-called  point-based  animations  (PBA)  technique,  will  be  applied  to  animate 
merged  imaging  data.  PBA  is  a  mesh-free  continuum  mechanics-based  framework  for  the 
animation  of  elastic,  plastic  and  melting  objects,  utilizing  point-based  volume  and  surface 
representations  derived  from  continuum  mechanics,  which  allows  the  specification  of 
common  material  properties  such  as  Young’s  Modulus  and  Poisson’s  Ratio. 


UD  colleagues  Rui  Hu,  Karl  Steiner,  and  Kenneth  Barner  developed  and 
implemented  a  physical  based  deformation  module  using  the  point-based  animation 
algorithm.  On  the  issue  of  simulating  deformable  objects,  the  physical-based  finite 
element  method  (FEM)  and  the  mass-spring  method  have  garnered  great  attention. 
However,  these  deformation  algorithms  share  some  common  issues:  1)  A  fine  mesh 
that  connects  nodes  is  required  for  both  FEM  and  mass-spring  model  simulation. 
This  leads  to  an  issue  that  the  mesh  must  be  constantly  updated  and  maintained 
when  model  topology  changes.  During  simulation,  arbitrary  element  decomposition 
such  as  cutting,  may  lead  to  ill-shaped  elements,  which  will  reduce  the  stability  for 
deformation.  2)  For  the  mass-spring  model,  one  has  to  fine-tune  the  spring 
parameters  so  that  the  model  can  deform  in  a  way  that  a  specific  soft  tissue  behaves; 
however,  fine-tuning  for  arbitrarily  shaped  models  is  rather  difficult.  To  avoid  these 
issues,  various  meshless  methods  have  emerged.  They  do  not  require  subdividing 
the  object  domain  into  subelements.  This  bypasses  the  cumbersome  meshing 
process.  The  deformation  model  is  only  comprised  of  mass  nodes,  which  are 
generated  by  sampling  the  object  volume  before  simulation.  Also,  as  this  approach 
follows  the  continuum  mechanics 
approach,  the  user  can  easily 
choose  the  corresponding  Poisson 
ratio  and  Young’s  Modulus  to 
control  the  behavior  of  an  object. 

Among  meshless  simulation 
approaches,  the  point  based 
animation  (PBA)  has  been  widely 
adopted  in  the  computer  graphics 
community.  The  UD  team 
successfully  implemented  the  PBA 
simulation  algorithm.  The  PBA 
algorithm  can  be  broken  into  five 
steps:  1)  Initialization;  2)  Using 
moving  least  squares  approximation  Fig.  33.  The  deformation  of  a  cube  sampled 
to  estimate  the  derivative  of  the  by  points  under  gravity.  Upper  left  and  right 

rows  are  fixed. 
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displacement  field;  3)  Updating  strains  and  stresses;  4)  Computing  forces;  5)  Time 
integration.  The  PBA  algorithm  works  on  a  group  of  points.  The  deformation  of  the 
model  is  controlled  by  the  movement  of  these  points.  Fig.  33  shows  the  deformation 
screenshots  when  a  cube  is  influenced  by  gravity.  The  cube  is  sampled  as  a  number 
of  points.  Two  rows  of  these  points  (top  left  and  right)  are  fixed  in  space,  while  the 
other  points  are  dragged  down  by  gravity. 

•  The  UD  team  used  PBA  to  update  surface  mesh.  The  point  movement  in  PBA 
simulation  is  used  to  model  the  physical  behavior  of  a  model  when  influenced  by 
external  forces.  To  map  the  point  movement  to  the  movement  of  surface  vertices, 
such  that  the  model  can  be  visualized  easily,  we  use  another  module  to  map  the 
displacement  field  of  points  to  the  counterpart  of  the  surface  mesh  vertices.  Each 
vertex  has  a  group  of  neighboring  points  specified  during  initialization.  Different 
points  have  different  weighting  values  on  the  vertex,  based  on  their  distance  to  the 
vertex.  The  displacement 
value  on  each  point  is 
weighted  using  this 
predefined  weighting  value 
and  then  sent  to  the  vertex. 

This  updated  surface  mesh  is 
used  for  visual  feedback.  Fig. 

34  exhibits  the  deformation 
of  a  tumor  model  falling  on  a 
hard  surface.  The  UD  team 
compared  the  behavior  of 
tumor  with  two  different 
Young’s  modulus 
parameters.  The  upper  row 
is  relatively  stiff,  while  the 
lower  row  is  very  soft. 

•  The  UD  team  simulated  deformable  bodies.  The  image-based  collision  detection 
and  contact  force  response  algorithm  follows  five  phases:  1)  broad  collision 
detection;  2)  preparation  of  viewport  setup;  3)  layered  depth  image  (LDI) 
generation;  4)  calculation  of  the  penetration  volume  and  its  gradient;  and  5) 
transition  of  the 
computed 
penalty  force  to 
the  deformation 
module.  In  the 
broad  collision 
detection  phase, 
the  axis-aligned 
bounding  box 
(AABB)  of  each 
object  is 


Fig.  35.  Virtual  palpation  of  a  tumor  model  using  a  rod-shaped 
instrument. 


Fig.  34.  The  deformation  of  a  soft  tumor  when 
falling  onto  a  virtual  hard  surface.  Top  row  -  stiff 
material,  bottom  row  -  soft  material. 
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updated.  If  there  is  an  intersection  of  two  AABBs,  a  fine  collision  detection  and 
response  scheme  is  conducted  by  analyzing  the  sampled  LDI  of  the  intersecting 
objects.  Since  LDI  contains  the  volume  overlapping  information,  the  interaction 
force  at  surface  vertices  can  be  easily  calculated.  In  Fig.  35,  the  simulation  scene  is 
illustrated  with  a  tumor  model.  The  development  of  this  approach  fulfills  the 
animation  section  in  Milestone  2.3.1  for  CT  data,  but  not  for  combined  PET/CT  data. 


Fig.  36.  3D  projection  in  SOFA  of  thorax  of 
patient  A,  showing  surface  skin  and  muscle 
beneath. 


Milestone  2.3.2.  SOFA-based  imaging  simulation  of  merged  pancreatic  cancer  FDG  PET/CT 
data  with  Phantom  haptic  feedback.  End  user  visualization  will  be  developed  with  the 
SOFA  open-source  real-time  simulation 
framework  as  an  environment  for 
medical  simulations.  End  user  haptic 
feedback  will  be  developed  with  the 
PHANTOM,  which  is  accurate  for  fine 
detail  work,  widely  used  by  many 
research  groups,  and  runs  on  a  variety  of 
computing  platforms. 


SOFA-based  imaging  simulation 
with  haptic  feedback  manipulation  of 
pancreatic  cancer  CT  data  from  TJU 
patient  A  is  underway.  The  visual 
models  in  Amira  have  been  converted 
into  surface  meshes  and  imported  in 
a  SOFA  simulation  scene  (Fig.  36). 
The  images  in  SOFA  are  currently 
visual  only  as  more  work  is  being 
done  to  generate  meshes  for  collision 
models  and  force  field  models.  The 
image  of  patient  A  in  SOFA  with  the 
skin  and  muscle  removed  also  shows 
the  liver  partly  transparent  (Fig.  37). 
The  models  being  transferred  from 
Amira  to  SOFA  have  a  high  polygon 
count  at  the  moment.  Reducing  the 
polygons  is  not  difficult  but  more 
delicate  structures  in  the  object  will 
be  lost.  Since  the  point  based 
animation  and  finite  element  method 
are  dependent  on  how  many  points 
are  used  to  represent  each  object, 
due  consideration  will  need  to  be 
made  when  deciding  on  an 
acceptable  polygon  count  for  the 
collision  and  forcefield  models. 
Further  work  will  need  to  be  done  to 


Fig.  37.  3D  projection  in  SOFA  of  internal 
anatomy  of  patient  A,  showing  ribs  and 
organs.  The  liver  is  partially  transparent: 
bones  (white),  liver  (translucent  gray), 
intestines  (pink),  pancreas  (gold), 
duodenum  (lavender),  aorta  (red),  stomach 
(olive),  diaphragm  (rust). 
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decide  on  an  acceptable  number  of  polygons  to  improve  performance  accurately 
represents  the  organs. 

•  A  new  method  for  haptic  interaction  using  the  Linear  Compliance  Problem  (LCP) 
is  being  tested.  This  new  method  has  alleviated  the  tendency  for  the  haptic  object  to 
become  stuck  to  the  model,  in  part  by  allowing  the  collision  response  to  be 
separately  calculated  from  the  physical  deformations.  Also  LCP  allows  different 
regions  of  the  object  to  behave  differently  upon  contact.  Fig.  38A  shows  the 
pancreas  (gold)  before  interacting  with  the  haptic  probe  (grey).  Fig.  38B  shows  the 
haptic  probe  interacting  with  a  region  of  the  pancreas  designated  to  be  soft  and 
easily  deformed,  such  as  interacting  with  a  necrotic  region  of  a  tumor.  In  Fig.  38C, 
the  haptic  object  is  interacting  with  a  stiff  area,  simulating  the  stiff  regions  of  a  tumor 
mass.  Finally,  Fig.  38D  shows  the  haptic  object  interacting  with  a  firm  yet  pliable 
region  that  could  be  representative  of  the  normal  pancreatic  tissue.  The  values  used 
for  contact  must  be  tuned  by  hand,  requiring  surgeon  feedback  on  how  stiff  or  soft 
certain  organs  feel.  Work  is  ongoing  to  incorporate  other  deformable  objects  in 
close  proximity  to  each  other.  These  steps  achieve  part  of  Milestone  2.3.2  for  CT 
data,  but  not  for  combined  PET/CT  data. 


B 


D 

Fig.  38.  3D  projection  in  SOFA  of  patient  A  pancreas  (gold),  with  a  haptic  probe  (grey).  A: 
pancreas  before  interacting  with  the  haptic  probe.  B:  probe  interacting  with  soft,  necrotic  region. 
C:  probe  interacting  with  stiff  active  tumor.  D:  probe  interacting  with  normal  remaining 
pancreas. 


£■*  mk 
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Task  2.4.  Human  perceptual  evaluation  by  experienced  surgeons  (Lead:  Kairys) 

Milestone  2.4.1.  Evaluation  by  6  experienced  surgeons  of  3D  animations  of  merged 
pancreatic  cancer  FDG  PET/CT  data:  assessment  ±  PET  images  (enhanced  spatial 
understanding).  Experienced  TJU  surgeons  (n=6)  will  qualitatively  evaluate  the  3D 
interface  for  ease  of  use  and  ability  to  intuitively  manipulate  and  adjust  the  image.  Users 
will  qualitatively  assess  whether  the  addition  of  the  3D  CT  reconstruction  and  merged 
FDG/PET  data  enhances  their  spatial  understanding  of  tumor  location,  enhances  their 
understanding  of  the  extent  of  pathology,  and  whether  this  display  enhances  their  ability 
to  plan  a  surgical  resection  for  lesions  of  different  sizes  and  locations  as  compared  to 
traditional  CT  images  alone. 

•  TJU  IRB  and  HRPO  approved  the  evaluation  questionnaire  for  3D  animations. 

The  questionnaire  follows. 

3-D  Volumetric  Imaging  -  Evaluation  by  Surgeons 


Case  Identifier 

Image  Evaluated:  (A,B,C,D...) 

Version:  (1,2,3...)  _ 

(circle  one) 

Visual  Effects 


Legend 

1  =  strongly  disagree 

2  =  disagree 

3  =  neither 
agree/disagree 

4  =  agree 

5  =  strongly  agree 


1.  The  display  hardware  is  easy/comfortable  to  use. 

2. 1  found  it  easy  to  manipulate/re-position  the  image. 

3.  The  organs/structures  are  accurately  represented  (accuracy  of 
segmentation). 

4.  Colors/textures  are  appropriate  (accuracy  of  rendering). 

5. 1  am  satisfied  with  the  level  of  detail  that  is  presented. 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 

1  2  3  4  5 
1  2  3  4  5 


6.  The  model  provides  me  with  adequate  reference  to  surrounding 
structures. 


1  2  3  4  5 
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7.  The  overall  3-dimensional  image  appears  realistic  (matches  what  I  1  2  3  4  5 

expect  to  see  in  the  OR). 

Usefulness 

8.  By  simply  viewing  the  image(s)  in  the  3-D  model,  I  get  a  better  1  2  3  4  5 

understanding  of  the  tumor  and  its  relationship  to  the 

surrounding  organs. 

9-  By  simply  viewing  the  image(s)  in  the  3-D  model,  my  plan  for  how  1  2  3  4  5 

to  approach  this  tumor  changed,  (as  compared  to  traditional  CT 
images). 

10.  By  overlaying  the  PET  data  in  the  3-D  model,  I  get  a  better  1  2  3  4  5 

understanding  of  the  tumor  and  its  relationship  to  the 

surrounding  organs. 

11.  By  overlaying  the  PET  data  in  the  3-D  model,  my  plan  for  how  to  1  2  3  4  5 

approach  to  this  tumor  changed  (as  compared  to  traditional  CT 

images). 

How  could  the  visual  appearance  of  this  model  be  improved? 


Applicability 


12.1  would  want  to  use  this  3-D  image  to  plan  an  operation  for  a 
patient  with  this  specific  tumor. 


1  2  3  4  5 


13.1  would  want  to  use  this  3-D  image  with  PET  overlay  to  plan  an 
operation  for  a  patient  with  this  specific  tumor. 


1  2  3  4  5 


14. 1  would  like  to  have  this  system  available  to  me  in  the  OR,  for 
reference  during  an  actual  operation. 


1  2  3  4  5 


15.1  believe  that  this  system  would  help  residents/assistants  better 
prepare  for  the  operation. 


1  2  3  4  5 
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How  could  the  usefulness  of  this  model  be  improved? 


Other  comments 


•  Six  surgeons  from  Thomas  Jefferson  University  Hospital  volunteered  to  evaluate 
version  1.2,  with  standard  CT,  and  1.3,  with  contrast-enhanced  CT,  of  the  Patient  A 
thoracic  visualization.  These  surgeons  varied  from  general  surgery  to  thoracic 
surgery.  The  surgeons  were  shown  how  to  manipulate  the  image,  especially  how  to 
strip  off  overlaying  tissues  and  organs  overlaying  the  pancreas.  The  average  scores 
for  each  of  the  15  questions  asked  for  versions  1.2  and  1.3  are  shown  in  Fig.  39abc. 
Averaging  surgeon  responses  to  each  of  the  15  questions,  version  1.2  scored  3.6  ± 
1.3,  while  version  1.3  received  a  score  of  4.4  ±  0.8.  The  overall  averages  of  the 
questionnaire  illustrate  greater  appreciation  of  version  1.3,  while  the  standard 
deviations  reveal  greater  consensus  in  evaluating  version  1.3.  The  set  of  scores  for 
1.3  were  significantly  higher  than  for  1.2,  according  to  a  paired  t-test. 
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Average  Score 


a)  accuracy 


1 .  The  display  hardw  are  is 
easy/comfortable  to  use 

2.  I  found  it  easy  to 
manipulate/re-position  the 
image 

3.  The  organs/structures  are 
accurately  represented 
(accuracy  of  segmentation), 

4.  Colors/textures  are 
appropriate  (accuracy  of 
rendering) 

5.  I  am  satisfied  w  ith  the  level 
of  detail  that  is  presented. 

6.  The  model  provides  me  with 
adequate  reference  to 
surrounding  structures 

7.  The  overall  3D  image 
appears  realistic  (matches 
w  hat  I  expect  to  see  in  OR) 


□  Version  1.2  aversion  1.3 


Fig.  39a.  Average  scores  for  version  1.2  and  1.3  of  the  visualization,  with  error  bars,  for 
questions  on  a)  the  ease  of  use  and  accuracy  of  models. 
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b)  understanding  00 


Average  Score 

2.0  3.0  4.0  5.0 


8.  By  simply  viewing  the 
image(s)  in  the  3D  model,  I  get  a 
better  understanding  of  the 
tumor  and  its  relationship  to  the 
surrounding  organs. 


9.  By  simply  viewing  the 
image(s)  in  the  3D  model,  my 
plan  for  how  to  approach  this 
tumor  changed,  (as  compared 
to  traditional  CT  images) 


10.  By  overlaying  the  PET  data 
in  the  3D  model,  I  get  a  better 
understanding  of  the  tumor  and 
its  relationship  to  the 
surrounding  organs. 


1 1 .  By  overlaying  the  PET  data 
in  the  3D  model,  my  plan  for 
how  to  approach  this  tumor 
changed,  (as  compared  to 
traditional  CT  images) 


□  Version  1.2  □  Version  1.3 


Fig.  39b.  Average  scores  for  version  1.2  and  1.3  of  the  visualization,  with  error  bars,  for 
questions  on  b)  whether  viewing  these  models,  with  or  without  PET  changed  their 
understanding  of  the  tumor. 
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c)  usefulness 


1.0 


Average  Score 

2.0  3.0  4.0  5.0 


12.  I  w  ould  w  ant  to  use  this  3D 
image  to  plan  an  operation  for 
a  patient  w  ith  a  specific  tumor 


1 3.  I  w  ould  w  ant  to  use  this  3D 
image  w  ith  PET  overlay  to  plan 
an  operation  for  a  patient  w  ith 
a  specific  tumor 


14.  I  w  ould  like  to  have  this 
system  available  to  me  in  the 
OR,  for  the  reference  during 
an  actual  operation 


15.  I  believe  that  this  system 
would  help  residents/assistant 
better  prepare  for  the 
operation. 


□  Version  1.2  □  Version  1.3 


Fig.  39c.  Average  scores  for  version  1.2  and  1.3  of  the  visualization,  with  error  bars,  for 
questions  on  c)  the  usefulness  of  the  3D  system. 


Milestone  2.4.2.  Evaluation  by  6  experienced  surgeons  of  haptic  manipulation  of  merged 
pancreatic  cancer  FDG  PET/CT  data:  assessment  ±  PET  images  (enhanced  spatial 
understanding).  Experienced  TJU  surgeons  (n=6)  will  utilize  the  haptic  interface  to 
“palpate”  the  pancreas  and  region  of  the  tumor.  A  qualitative  assessment  will  be 
performed  to  determine  the  realism  of  the  experience,  and  to  determine  optimal 
assignment  and  adjustment  of  the  reconstructed  tissue’s  Young’s  Modulus  and  Poisson’s 
Ratio  to  best  represent  the  conditions  experienced  during  actual  surgery.  The  benefit  of 
overlaying  the  FDG  PET  data,  allowing  refinement  of  the  haptic  experience  by  further 
adjustment  of  the  reconstructed  tissues’  properties,  will  also  be  assessed.  The  addition  of 
enhanced  visual  feedback  (such  as  partial  transparency)  or  enhanced  haptic  feedback 
(such  as  a  vibratory  sensation  when  palpating  over  areas  of  increased  FDG  PET  activity) 
will  be  assessed  to  determine  if  these  techniques  subjectively  improve  the  surgeons’ 
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perception  of  the  location  and  extent  of  disease.  These  techniques  will  be  evaluated  for 
lesions  of  different  sizes  and  locations,  including  small  lesions  that  may  otherwise  not  be 
easily  palpable  in  real  life  situations. 

•  TJU  IRB  and  HRPO  approved  the  evaluation  questionnaire  for  3D  animations 
including  haptic  manipulation.  The  questionnaire  follows. 

3-D  Haptic  Imaging  -  Evaluation  by  Surgeons 


Case  Identifier 


Image  Evaluated:  (A,B,C,D...) 


Version:  (1,2,3  .  ) 


Legend 

1  =  strongly  disagree 

2  =  disagree 

3  =  neither 
agree/disagree 

4  =  agree 

5  =  strongly  agree 


Haptic  ("Touch  and  Feel")  Effects 


(circle  one) 


l.The  haptic  interface  devices  are  easy  to  use. 


1  2  3  4  5 


2. The  haptic  interface  devices  provide  an  adequate  range  of 
movement. 


1  2  3  4  5 


3. 1  am  satisfied  with  how  I  interact  with  the  model. 


1  2  3  4  5 


4.  When  manipulating/touching  the  model,  the  displacement  of  the  1  2  3  4  5 

normal  structures  is  realistic. 

5.  When  manipulating/touching  the  model,  the  displacement  of  the  1  2  3  4  5 

tumor  is  realistic. 


6.  When  manipulating/touching  the  model,  the  texture  of  the 
normal  structures  is  realistic. 


1  2  3  4  5 


7.  When  manipulating/touching  the  model,  the  texture  of  the  tumor  1  2  3  4  5 

is  realistic. 


8.  When  manipulating/touching  the  model,  the  relative  movement  1  2  3  4  5 

of  the  organs  is  realistic. 
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9- When  manipulating/touching  the  model,  the  vibration  felt  over  1  2  3  4  5 

the  tumor  gave  me  a  better  understanding  of  malignant  tissue  vs. 
clear  margins. 

Usefulness 


10.  By  touching  and  feeling  the  model,  I  get  a  better  understanding  1  2  3  4  5 

of  the  tumor  and  its  relationship  to  the  surrounding  organs. 

11.  By  touching  and  feeling  the  model,  I  can  do  a  better  job  planning  1  2  3  4  5 

my  approach  to  this  tumor. 

12.  By  overlaying  the  PET  data  and  hap tically  interacting  with  the  1  2  3  4  5 

model,  I  get  a  better  understanding  of  the  tumor  and  its 

relationship  to  the  surrounding  organs. 

13.  By  overlaying  the  PET  data  and  hap  tically  interacting  with  the  1  2  3  4  5 

model,  my  plan  for  how  to  approach  to  this  tumor  changed,  (as 

compared  to  traditional  CT  images). 


How  could  the  touch  and  feel  of  this  model  be  improved? 


Applicability 


14. 1  would  want  to  use  this  3-D  image  to  plan  an  operation  for  a  1  2  3  4  5 

patient  with  this  specific  tumor. 

15.1  would  want  to  use  this  3-D  image  with  PET  overlay  to  plan  an  1  2  3  4  5 

operation  for  a  patient  with  this  specific  tumor. 


16. 1  would  want  to  use  this  3-D  image  with  PET  overlay  and  haptic  1  2  3  4  5 

interaction  to  plan  an  operation  for  a  patient  with  this  specific 
tumor. 


17.1  would  like  to  have  this  system  available  to  me  in  the  OR,  for 
reference  during  an  actual  operation. 


1  2  3  4  5 
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18.1  believe  that  this  system  would  help  residents/assistants  better  1  2  3  4  5 

prepare  for  the  operation. 

How  could  the  usefulness  of  this  model  be  improved? 


Other  comments 


The  questionnaire  will  be  administered  after  the  haptic  model  satisfies  the 
investigators  as  a  realistic  presentation. 
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C.  KEY  RESEARCH  ACCOMPLISHMENTS 

•  Molecular  dynamics  calculations  implied  that  EGF  binding  to  the  extracellular 
domain  of  EGFR  (sEGFR)  disrupts  auto-inhibitory  intramolecular  interactions  by 
increasing  the  distance  between  the  so-called  “dimerization  arm”  and  domain  IV. 

•  Molecular  dynamics  calculations  implied  that  EGFR  dimerization  is  more  favorable 
for  one  EGF/EGFR  complex  plus  one  empty  EGFR.  Such  asymmetrical  dynamics  in 
EGFR  provide  a  structural  rationale  for  the  observed  negative  cooperative  effects  in 
EGFR  activation. 

•  Molecular  dynamics  calculations  implied  that  single  loop  EGF  fragments  would  be 
insufficient  to  activate  EGFR  or  initiate  internalization. 

•  Direct  measurements  of  cellular  uptake  of  two  different  EGF  fragments,  EGF20-31 
and  EGF32-48,  as  well  as  KRAS  PNA-EGF32-48,  agreed  with  the  computer 
predictions  of  nonspecific  uptake. 

•  3D  visualization  of  the  thorax  of  a  patient  with  pancreatic  cancer  was  evaluated  as 
useful  to  surgeons  and  applicable  in  the  clinic. 
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E.  CONCLUSIONS 

The  epidermal  growth  factor  receptor  (EGFR)  is  a  member  of  the  ErbB  receptor  tyrosine 
kinase  (RTK)  family  that  plays  a  pivotal  role  in  cell  proliferation,  differentiation  and 
migration.  Aim  1  explored  the  possibility  that  EGFR  might  serve  as  a  gateway  into  cells  for 
mRNA  imaging  agents.  While  x-ray  structures  have  provided  insight  into  EGFR  function, 
the  conformational  pathway(s)  of  ligand-dependent  activation  are  not  well  understood. 
Using  molecular  dynamics  (MD)  techniques  we  found  that  EGF  binding  to  the  extracellular 
domain  of  EGFR  (sEGFR)  disrupts  auto-inhibitory  intramolecular  interactions  by  increasing 
the  distance  between  the  so-called  “dimerization  arm”  and  domain  IV.  To  investigate  early 
ligand  binding  effects  on  sEGFR  dimers,  we  constructed  a  singly  ligated  sEGFR  dimer  for 
simulation.  MD  simulations  of  the  singly  ligated  sEGFR  dimer  exhibited  asymmetrical 
motion  in  each  monomer  and  a  decrease  in  the  size  of  the  second  EGF  binding  pocket  by 
10  A.  Free  energy  calculations  indicated  that  asymmetrical  dimerization  with  only  one  EGF- 
sEGFR  monomer  is  more  energetically  favorable  than  dimerization  of  two  ligated  sEGFR 
monomers  by  2.88  kcal*mol-l.  To  see  if  asymmetrical  motion  can  be  extended  to  ErbB 
heterodimers,  we  simulated  a  model  sEGFR-ErbB2  dimer.  This  heterodimer  also  exhibited 
asymmetrical  motion.  These  results  suggest  that  asymmetrical  dynamics  in  sEGFR  and 
other  ErbB  family  members  play  an  essential  role  in  receptor  activation  and  cooperativity. 
From  this  study,  we  inadvertently  realized  a  new  potential  target  for  therapeutic  inhibition 
of  EGFR  activation. 

On  the  other  hand,  MD  calculations  implied  that  single  loop  EGF  fragments  would  be 
insufficient  to  activate  EGFR  or  initiate  internalization.  Direct  measurements  of  cellular 
uptake  of  two  different  EGF  fragments,  EGF20-31  and  EGF32-48,  as  well  as  KRAS  PNA- 
EGF32-48,  agreed  with  the  computer  predictions  of  nonspecific  uptake.  Hence,  EGFR 
constitutes  an  unlikely  candidate  to  serve  as  a  gateway  into  cells  for  mRNA  imaging  agents, 
as  intended  in  Aim  1.  IGF1R  continues  to  serve  as  a  useful  gateway,  as  we  found  previously 
for  breast,  pancreatic,  and  prostate  cancer  cells.  Concurrent  work  supported  by  NIDA 
illustrated  that  the  /x-opioid  receptor  also  functions  as  a  gateway  for  mRNA  imaging  agents. 

Surgical  determination  of  whether  to  proceed  with  a  pancreatectomy  requires  3D  spatial 
awareness  for  many  structures  of  interest  (tumor,  vasculature,  pancreas,  and  spleen)  and 
how  the  tumor  impacts  upon  each.  Typically  the  decision  and  planning  steps  are  based 
upon  two  dimensional  (2D)  medical  images  of  the  patient  from  a  variety  of  sources 
including  computed  tomography  (CT),  magnetic  resonance  imaging,  ultrasound, 
photoacoustic  imaging,  or  positron  emission  tomography  (PET).  Aim  2  sought  to 
determine  whether  three-dimensional  (3D)  presentation  of  PET/CT  data  would  benefit 
surgeons  preparing  for  a  procedure.  3D  rendering  and  co-registration  of  PET  and  CT 
thoracic  scans  from  a  patient  with  a  ductal  pancreatic  mass  were  created,  then  evaluated  by 
6  experienced  surgeons  by  answering  a  questionnaire.  Contrast-enhanced  CT  fused  with 
PET  provided  a  significantly  more  lifelike  presentation  than  standard  CT,  increasing  the 
usefulness  of  the  presentation,  according  to  the  surgeons’  evaluations.  Hence,  3D 
simulation  of  a  specific  patient  from  fused  PET/CT  data,  the  goal  of  Aim  2,  could  be  a 
useful  preparatory  tool  for  surgeons  prior  to  making  the  first  incision.  A  haptic  model  of 
the  same  patient  is  being  developed,  and  will  be  evaluated  by  the  same  6  surgeons  once 
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the  texture  and  responsiveness  of  the  pancreas  are  satisfactory. 

So  what?  External  imaging  of  disease  gene  expression  in  live  patients  will  be  the  next  step 
in  our  research  path,  even  though  EGFR  lacks  the  properties  needed  for  a  gateway.  3D 
touch-and-feel  manipulation  of  patient  images  in  real  time  just  before  surgery  will  enable 
shorter  procedures  with  fewer  complications. 
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Abstract.  We  are  developing  agents  for  positron  emission  tomography  (PET)  imaging  of  cancer 
gene  mRNA  expression  and  software  to  fuse  mRNA  PET  images  with  anatomical  computerized 
tomography  (CT)  images  to  enable  volumetric  (3D)  haptic  (touch-and-feel)  simulation  of 
pancreatic  cancer  and  surrounding  organs  prior  to  surgery  in  a  particular  patient.  We  have 
identified  a  novel  ligand  specific  for  epidermal  growth  factor  receptor  (EGFR)  to  direct  PET 
agent  uptake  specifically  into  cancer  cells,  and  created  a  volumetric  haptic  surgical  simulation  of 
human  pancreatic  cancer  reconstructed  from  patient  CT  data.  Young’s  modulus  and  the  Poisson 
ratio  for  each  tissue  will  be  adjusted  to  fit  the  experience  of  participating  surgeons. 

Keywords.  Cancer,  haptic,  pancreas,  SOFA,  surgery,  tumor,  volumetric 


Introduction 

Surgery  involves  palpating  and  manipulating  tissues  in  the  operating  room  environment. 
However,  sophisticated  radiographic  systems  present  only  visual  images.  The  actual  assembly  of 
organs  of  a  particular  patient  must  now  be  imagined  by  the  surgeon  before  the  operation. 
Complications  that  were  not  anticipated,  such  as  bleeding  from  unusually  placed  arteries  or 
veins,  or  unusual  lesion  geometry,  lengthen  the  procedure,  placing  extra  stress  on  the  patient  and 
the  surgeons. 

One  solution  is  the  Simbionix  Procedure  Rehearsal  Studio™  program  for  the 
AngioMentor™  simulator  that  allows  surgeons  to  upload  patients’  actual  CT  angiogram  images 
and  then  perform  a  virtual  procedure,  complete  with  visual  and  haptic  feedback,  on  that  patient’s 
anatomy  [7],  But  the  Simbionix  system  only  uses  anatomical  data. 

We  hypothesize  that  our  fusion  of  genetic,  visual,  and  tactile  information  will  improve 
surgeons’  understanding  of  the  extent  of  disease  and  will  ultimately  permit  surgeons  to  better 
plan  operations  and  to  prepare  for  the  actual  pathology  found. 

1.  Methods 

1.1.  PET  Imaging  Agents  for  mRNA 

We  design  novel  EGFR  ligands  by  molecular  dynamics  using  Amber  10  [4]  force  fields, 
followed  by  haptic  sensing  of  the  kinetic  pathway  of  ligand  binding  to  the  receptor.  The 
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energetically  optimal  peptide  selected  above  is  extended  from  a  solid  phase  support  by  solid 
phase  coupling  of  Fmoc-L-amino  acid  monomers.  The  terminal  Cys  residues  are  cyclized.  A 
diethyleneglycol  spacer  is  then  coupled,  followed  by  the  mutant  KRAS2  G12D  PNA  sequence,  a 
second  di ethyleneglycol  spacer,  then  finally  a  fluorophore  or  a  DOTA  chelator  [8],  PNA 
mismatch  and  peptide  mismatch  controls  are  also  prepared.  The  finished  reporter-PNA-peptides 
are  cleaved  from  the  solid  support,  deprotected,  purified  by  reversed  phase  liquid 
chromatography,  and  analyzed  by  mass  spectroscopy  to  determine  whether  the  desired  sequence 
was  synthesized  correctly. 

1.2.  Volumetric  Images 

We  assembled  a  prototype  system  for  haptic  manipulation  of  volumetric  images  of  pancreatic 
cancer  gene  expression  and  anatomy.  Slicer3  [5],  Amira  5  (Visage  Imaging)  SOFA  [9]  and 
Phantom  Omni  manipulators  (SensAble)  are  installed  on  a  3DBoxx  8570  (Boxx  Technologies) 
with  8  computing  nodes  and  48  GB  of  RAM  to  support  an  Nvidia  5800  video  card  with  4  GB 
GPU. 

1.3.  Haptic  Manipulation 

A  pair  of  Phantom  Omni  manipulators  is  controlled  by  the  end-user  to  provide  translation  and 
rotation  input  information  for  surgery  instruments  in  the  virtual  reality  environment.  A  collision 
detection  algorithm  is  adopted  to  detect  the  penetration  between  surface  meshes  of  virtual  organs 
and  instruments.  If  a  mesh  controlled  by  the  haptic  device  is  involved  in  a  collision,  the 
corresponding  penetration  volume  is  sampled.  The  haptic  force  is  calculated  using  this  volume 
and  feedback  is  provided  for  the  haptic  device  to  give  the  end-user  realistic  feedback. 

To  render  variations  in  haptic  stiffness  properties  in  different  regions  of  the  target  tissue,  we 
strategically  place  key  vertices  over  the  surface  mesh.  Each  of  these  vertices  contains  an 
influence  radius.  As  the  virtual  instrument  enters  any  of  these  radiuses,  the  feedback  force  is 
appropriately  scaled.  The  proximity  of  the  surgery  instrument  to  the  vertex  and  a  predefined  base 
stiffness  value  control  the  scale  factor,  creating  distinct  regional  haptic  properties. 

2.  Results 

2.1.  Modeling  and  Testing  ofEGFR  Ligands  for  mRNA  PET  Imaging  Agents 

The  EGF  central  loop  fragment,  amino  acids  20-31,  CMYIEALDKYAC,  with  Cys-Cys  disulfide 
bridge  between  Cys20 
and  Cys31  (Fig.  1A), 
was  selected 

empirically  from  the 
crystal  structure  [1]. 

EGF  amino  acids  32- 
48, 


Figure  1.  Potential  EGF  ligands,  energy-minimized  by  molecular  dynamics  in  Chem3D.  A:  cyclized 
EGF20-31  selected  from  crystal  structure  [1];  B:  cyclized  EGFR-binding  EGF32-48  identified  by  cell 
binding  [2];  C:  EGFR-binding  peptide  identified  by  phage  display  [3]. 
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N  CW  G  YIGERCQ  YRDLK,  with  Cys-Cys 
disulfide  bridge  between  Cys33  and  Cys42  (Fig. 
IB),  was  selected  as  a  second  candidate  ligand, 
based  on  binding  to  EGFR  on  cells  [2].  GE11, 
YHWY GYTPQNVI,  an  EGFR-binding  peptide 
identified  by  phage  display  [3]  (Fig.  1C),  was 
selected  as  a  third  candidate  ligand. 

The  molecular  dynamics  docking  pathway  of 
EGF  into  the  ligand  binding  pocket  of  EGFR  was 
then  calculated  with  ZDOCK,  which  performs 
rigid  body  docking  calculations  and  determines 
their  relative  energies  in  implicit  water  with  the 
appropriate  dielectric  constant,  on  Salk  at  the 
Pittsburgh  Supercomputer  Center  (Fig.  2). 
Fluorescence  microscopy  and  flow  cytometry  revealed 
that  AlexaFluor  532-EGF20-31  fails  to  enter  cells  that 
overexpress  EGFR.  On  the  other  hand,  AlexaFluor 
532-GE1 1  enters  cells  whether  or  not  they  overexpress 
EGFR.  Fullsize  recombinant  EGF  failed  to  block 
uptake  of  Alexa  Fluor  532-GE1 1.  Chinese  hamster 
ovary  (CHO)  cells,  lacking  EGFR,  also  took  up 
AlexaFluor  532-GE1 1.  This  result  implies  that  GE1 1 
uptake  is  nonspecific.  However,  AlexaFluor  532- 
EGF32-48  was  taken  up  by  human  AsPCl  pancreatic 
cancer  cells  that  overexpress  EGFR,  examined  by 
confocal  fluorescence  microscopy  to  evaluate  binding. 

2.2.  Volumetric  Simulation  with  Haptic  Feedback 


Figure  2.  Graphic  image  of  EGF  docking  with  EGFR  from  an 
initial  20  A  out  of  the  ligand  binding  pocket  after  500  ps  of 
molecular  dynamics  energy  minimization,  using  ZDOCK  on 
the  Salk  64-bit  high  performance  computer  of  the  Pittsburgh 
Supercomputer  Center. 


Volumetric  simulation  with  haptic  feedback 
manipulation  of  pancreatic  cancer  CT  data  from 
deidentified  patient  A  (IRB  approved)  was  carried  out  using  SOFA 
(Fig.  3). 

We  observed  that  the  tissue  texture  parameters  do  not  yet  reflect  the 
actual  feel  of  each  organ  and  lesion.  The  pancreatic  tumor  was 
isolated  (Fig.  4).  Young’s  modulus  and  the  Poisson  ratio  will  be 
varied  in  a  search  for  typical  tumor  texture  and  feel. 


Figure  3.  3D  projection  of  thorax  of  patient  A. 
Advanced  pancreatic  tumor  (right)  extends  from  the 
head  of  the  pancreas  (left).  The  liver  and  kidneys  are 
also  shown,  with  the  scalpel  manipulating  a  kidney 
(bottom  right). 


PET  data  from  uptake  of  [18F]fluorodeoxyglucose  (FDG)  were  then 
overlaid  on  the  CT  data  for  patient  A  (Fig.  5).  Intense  FDG 
concentration  was  seen  in  the  tumor,  in  agreement  with  the 
aggressive  nature  of  the  pancreatic  malignancy. 


Figure  4.  SOFA-based  projection  of 
pancreatic  tumor  from  Fig.  3, 
separated  from  other  organs.  Scalpel 
tool  is  manipulated  for  touch-and- 
feel  variation. 
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Bladder  uptake 
reflects  normal 
excretion  of  FDG. 

The  next  phase  of 
haptic  development 
will  utilize  co¬ 
registered  PET/CT 
data  to  highlight 
aggressive  tumor 
tissue  for  excision. 

3.  Discussion 

3.1  EGFR  Ligands  for  mRNA  PET  Imaging  Agents 

EGF32-48  fragment  calculations  agreed  with  experimental  cellular  uptake,  as  well  as  non¬ 
binding  by  EGF20-3 1  fragment.  We  discovered  that  the  energies  of  the  entire  kinetic  docking 
pathway  could  be  calculated  in  a  single  run,  beginning  with  EGF  translated  20  A  out  of  EGFR. 
The  ZDOCK  force  fields  were  sufficient  to  draw  EGF  into  the  ligand  binding  pocket.  This 
insight  eliminates  the  need  for  separate  calculations  over  a  series  of  specific  separations. 

3.2.  Volumetric  Simulation  with  Haptic  Feedback 

We  also  observed  that  the  haptic  manipulators  felt  heavier  than  a  real  scalpel  or  retractor.  This 
suggests  a  future  need  for  wireless  glove  manipulators.  The  next  phase  of  haptic  development 
will  utilize  co-registered  PET/CT  data  to  highlight  aggressive  tumor  tissue  for  excision. 


Figure  5.  3D  projection  of  fused  PET/CT  data  at  5  mm  resolution  of  patient  A.  CT  images  of  bones  and 
organs  are  light,  PET  images  are  dark. 
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A  Generalized  Haptic  Feedback  Approach  for  Arbitrarily  Shaped  Objects 

Rui  HU,  Kenneth  E.  BARNER,  and  Karl  V.  STEINER 
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Abstract.  In  surgery  procedures,  haptic  interaction  provides  surgeons  with  indispensable 
information  to  accurately  locate  the  surgery  target.  This  is  especially  critical  when  visual 
information  cannot  provide  sufficient  information  and  tactile  interrogation,  such  as  palpating  some 
region  of  tissue,  is  required  to  locate  a  specific  underlying  tumor.  However,  in  most  current  surgery 
simulators,  the  haptic  interaction  model  is  usually  simplified  into  a  contact  sphere  or  rod  model, 
leaving  arbitrarily  shaped  intersection  haptic  feedback  between  target  tissue  and  surgery  instrument 
less  unreliable.  In  this  paper,  a  novel  haptic  feedback  algorithm  is  introduced  for  generating  the 
feedback  forces  in  surgery  simula-tions.  The  proposed  algorithm  initially  employs  three  Layered 
Depth  Images  (LDI)  to  sample  the  3D  objects  in  X,  Y  and  Z  directions.  A  secondary  analysis  scans 
through  two  sampled  meshes  and  detects  their  penetration  volume.  Based  on  the  principle  that 
interaction  force  should  minimize  the  penetration  volume,  the  haptic  feedback  force  is  derived 
directly.  Additionally,  a  post-processing  technique  is  developed  to  render  distinct  physi-cal  tissue 
properties  across  different  interaction  areas.  The  proposed  approach  does  not  require  any  pre¬ 
processing  and  is  applicable  for  both  rigid  and  deformable  objects. 

Keywords.  Haptic  Feedback,  Surgery  Simulation,  Penetration  Volume,  Layered  Depth  Images. 

Introduction 

For  many  years,  haptic  rendering  has  garnered  great  attention  in  multiple  fields,  including 
surgery  simulation,  virtual  reality  and  computer  games.  Haptic  feedback  in  virtual  reality  is 
coupled  with  visual  information  to  improve  the  simulation  environment  realism.  In  surgery 
simulation  applications,  haptic  rendering  can  serve  to  provide  surgeons  with  useful  properties, 
such  as  tissue  stiffness  information,  so  that  the  user  will  gain  a  better  understand  the  surgery 
situation  through  touching  and  palpation.  For  example:  recently  in  [1],  Laura  et  al.  explored  the 
possibility  of  adding  an  artificial  haptic  response  to  Phacoemulsification  cataract  surgery 
simulation  to  enhance  training  effectiveness.  However,  the  complex  geometry  shape  of  3D 
objects,  especially  deformable  surgery  objects,  makes  haptic  rendering  a  computationintensive 
task.  In  early  research,  to  alleviate  computational  complexity,  researchers  simplified  the  surgery 
instrument  into  a  point.  Zilles  et  al.  [2]  successfully  achieved  realtime  pointbased  haptic 
feedback  with  polygon  models.  In  [3],  Daniel  et  al.  simplify  multiple  surgery  instruments  by 
using  them  as  onepoint  haptic  models.  In  [4],  Matthias  et  al.  apply  the  haptic  feedback  algorithm 
by  Ruspini  et  al.  [5]  to  a  hysteroscopy  simulator  system,  realizing  both  the  single-point  and 
multiple-point  interaction  in  the  simulator.  In  their  organ  exclusion  simulation,  Yoshihiro  et  al. 

[8]  track  each  fmg  ertip  as  points  and  generate  the  haptic  force  for  each  point  using  a 
precomputed  stiffness  matrix  based  on  the  finite  element  method  (FEM).  This  algorithm  is 
accurate,  but  relatively  slow  and  cannot  cope  with  topology  changes.  As  a  comparison,  the 
Rensselaer  Polytechnic  Institute  group  members,  such  as  Suvranu  De  and  YiJe  Lim  focus  on 


W81XWH-09-1-0577  Final 


43 


Wickstrom,  Eric 


using  meshfree  methods  in  many  applications  [6,  16].  Beyond  the  point-based  haptic  model,  a 
sphere-based  model  is  also  employed  in  surgery  simulation  due  to  its  simplicity.  In  [7],  Peter  et 
al.  model  the  tip  of  a  drilling  instrument  as  a  sphere.  For  researchers  addressing  the  arbitrarily 
shaped  haptic  model  interaction  case,  they  typically  employ  methods  based  on  the  penetration 
depth  between  two  triangle  meshes  [9,  10].  In  [11],  Redon  et  al.  apply  a  real-time  penetration 
depth  calculation  method  for  complex  and  non-convex  meshes.  However,  this  algorithm  suffers 
from  some  penetration  distance  and  direction  approximation  issue.  In  [12],  Fisher  et  al.  consider 
calculating  the  penetration  volume  as  “The  most  complicated  yet  accurate  method”  to  describe 
the  extent  of  intersection.  Following  this  idea,  Weller  et  al.  [13]  proposes  a  new  data  structure, 
the  “Inner  Sphere  Tree”  to  estimate  the  penetration  volume.  However,  this  algorithm  relies 
heavily  on  the  bounding  volume  hierarchy  (BVH);  therefore,  it  is  not  applicable  for  deformable 
objects.  In  [14],  Faure  et  al.  apply  Layered  Depth  Image  (LDI)  to  calculate  the  penetration 
volume  for  the  purpose  of  deformation  simulation,  but  the  LDI  generation  method  employed  was 
too  slow  for  real-time  haptic  response. 

To  achieve  accurate  haptic  feedback  for  deformable  objects,  we  advance  the  LDI  method  to 
accurately  and  efficiently  compute  the  penetration  volume.  Three  new  LDIs  are  generated  for 
each  new  haptic  frame;  according-ly,  the  proposed  algorithm  is  well  suited  for  soft,  deform-able 
objects.  We  resort  to  a  post-processing  technique  to  render  tissue  stiffness  for  organs  with 
different  materials  at  different  locations,  such  as  a  relatively  stiffer  tumor  tissue  surrounded  by 
healthy,  soft  body  tissue. 

1.  Methods 

1.1.  Haptic  force  generation 

1.1.1.  Layered  depth  image  and  penetration  volume  generation 

Layered  Depth  Image  (LDI)  was  first  introduced  as  a  rendering  technique  [17],  The  LDI 
generation  procedure  is  essentially  a  sampling  process  operating  on  the  triangle  mesh  volume 
through  a  user-defined  viewport  and  resolution.  Suppose  we  shoot  a  ray  at  the  center  of  each 
pixel  in  the  viewport  and  the  ray  direction  is  orthogonal  to  the  view  plane.  Each  time  the  ray 
intersects  with  the  target  triangle  mesh,  a  sequence  of  intersected  fragment  information  is  stored 
in  the  LDI.  A  generated  LDI  contains  three  types  of  information  at  all  sampled  fragments, 
namely:  (1)  the  depth  values  through  the  viewing  direction,  (2)  which  object  this  fragment 
belongs  to,  and  (3)  the  object  normal  at  this  fragment  in  the  view  direction. 

LDI  greatly  facilitates  determination  of  the  penetration  volume  because,  by  sequentially 
scanning  the  normal  directions  and  object  numbers  of  all  fragments  at  a  pixel,  we  can  easily  find 
the  fragment  pair  where  two  objects  penetrate.  This  is  a  result  of  the  fact  that  if  we  follow  a  ray, 
every  fragment  the  ray  encounters  with  a  normal  opposing  the  ray  direction  means  that  the  ray  is 
entering  a  mesh,  while  a  fragment  with  a  normal  aligned  with  the  ray  direction  indicates  the  ray 
is  leaving  a  mesh.  By  multiplying  a  specific  penetration  length  with  a  pixel  area  dependent  on 
resolution,  we  approximate  the  penetration  volume  at  this  pixel.  Figure  1  illustrates  this  process 
of  penetration  volume  generation.  In  Fig.  l-(a),  the  bounding  volume  intersections  of  two  objects 
are  detected.  The  left  column  of  the  green  box  represents  the  view  plane.  The  volume  is 
rasterized  and  sampled  as  shown  in  Fig.  l-(b)  and  l-(c).  The  generated  LDI  is  shown  in  Fig.  1- 
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(d),  were  the  arrow  direction  indicates  the  normal  direction;  the  color  represents  the  object  that  is 
sampled.  We  store  the  fragments  and  then  pick  up  collision  pairs  in  each  pixel,  as  shown  in  Fig. 
l-(e).  Finally,  in  Fig.  l-(f),  the  length  of  the  penetration  section  is  calculated  at  each  pixel. 
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Figure  1.  An  illustration  of  calculating  intersection  volume  from  a  generated  Layered  Depth  Image  (LDI) 


1.1.2.  Efficient  LDI  calculation 

In  the  computer  graphics  area,  researchers  such  as  Everitt,  Bavoil  and  Myers  have  been 
investigating  ways  to  improve  the  LDI  generation  efficiency  for  years  [18-20],  Recently,  Liu  et  al. 
[15]  show  that  the  Compute  Unified  Device  Architecture  (CUDA)-based  LDI  generation  method 
yields  the  best  performance.  In  our  implementation,  we  have  adopted  a  similar  algorithm  for  LDI 
generation.  This  is  due  to  its  excellent  performance,  and  the  fact  that  CUDA  made  parallel 
penetration  length  query  and  operations  on  penetration  volume  available  on  GPUs. 

1.1.3.  Volume  based  haptic  force 

Similar  to  [14],  we  define  the  potential  energy  as:  E  =  14  kV2,  where  k  is  a  user-defined 
parameter  to  scale  this  energy,  and  V  is  the  penetration  volume.  Suppose  the  calculated  LDI  is 
along  the  Z  direction.  Now,  we  derive  the  final  haptic  force  along  a  certain  viewing  direction. 
Given  the  assumption  that  the  haptic  force  serves  to  minimize  the  intersection  volume  of  two 
objects,  the  interaction  force  in  the  viewing  direction  (Z  direction)  at  a  given  coordinate  (xa,  ya, 
za,)  is: 


dE  I 

dz  |  (xa.ya) 


C  xa.ya) 


(1) 
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If  we  take  the  haptic  force  direction  into  consideration,  i.e.  if  the  haptic -related  mesh  is  pushed  from 
right  to  left,  the  haptic  force  should  follow  the  negative  X-axis  direction.  Also,  if  no  haptic  model  is 
involved,  no  haptic  force  should  be  generated.  The  direction  factor  should  be: 

D  =0L-0R,  (2) 

where  and  stand  for  the  force  direction  influence  from  the  left  and  the  right  fragment.  We  now  have: 

{1,  fragment  i  belongs  to  haptic  object 
0,  fragment  i  belongs  to  non-haptic  object 

By  combining  the  factors  together,  the  haptic  force  in  the  Z  direction  is  the  integration  of  forces  at 
all  penetration  volume  boundaries: 

fz  =  fx  4  ~kv  ^Ddydx  (4) 

If  we  write  Equation  (4)  into  a  discrete  form  for  comput-er-based  calculation,  with  every  unique  X 
and  Y  standing  for  a  pixel  on  the  viewport,  the  term  dV/dz  automatically  becomes  1 .  Therefore  we 
have: 

Fz  =  T.x'Zy-WD  =  -k'Zx'ZyVD  (5) 

Equation  (5)  indicates  that  the  final  haptic  feedback  force  in  one  direction  is  simply  the  sum  of  all 
penetration  volume  weighed  by  force  direction.  This  greatly  facilitates  the  algorithm  programming. 
By  repeating  this  process  two  more  times  for  the  X  and  Y  directions,  we  generate  the  haptic  force, 
which  can  be  sent  to  the  haptic  device  directly  or  used  for  post-processing.  We  have  also  developed 
an  algorithm  to  infer  the  LDI  in  the  Y  and  Z  viewing  direction  from  the  LDI  in  X  direction.  This 
technique  further  reduces  the  overall  computation  time  by  about  one  half  to  two  thirds. 

1.2.  Haptic  force  post-processing 

In  order  to  render  the  distinct  feedback  forces  at  different  locations,  we  apply  the  following  post¬ 
processing  method  called  local  stiffness  vertex  method  to  the  calculated  haptic  force: 

1 .  At  simulation  initiation,  load  a  series  of  vertex  information  into  memory.  The  information  of  each 
vertex  i  consists  of  three  pieces  of  information:  (1)  the  vertex  number,  Nt;  (2)  the  influence  radius  of 
this  vertex,  R(;  and  (3)  the  stiffness  parameter  of  this  vertex,  S,. 

2.  During  simulation,  keep  monitoring  the  position  of  the  instrument  center  Pc. 

3.  The  distance  between  the  instrument  center  and  any  stored  vertex  i,  Dci  is  compared  with  the 
vertex  radius  /?,. 

4.  If  any  Dci  is  smaller  than  Rh  we  use  the  following  equation  to  scale  the  haptic  force: 
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Fpost  —  F  ZvO 


Si) 


(6) 


where  a,(x)  is  a  monotonic  continuous  function,  with  a/0)  =  l,  a,(l )=  1/  S,.  It  is  essentially  an 
influence  kernel  describing  how  the  stiffness  parameter  decreases  from  the  center  of  the  vertex  to 
its  surrounding  area. 


We  have  to  note  that  the  aforementioned  post-processing  algorithm  may  not  be  effective  on  large 
instruments  since  a  center  cannot  describe  the  real  shape  of  a  large  instrument  sufficiently.  We 
are  finalizing  improvements  to  the  algorithm  that  dynamically  add  stiffness  properties  to 
penetration  volumes.  This  algorithm  will  support  large,  arbitrary  instruments. 


2.  Results 


The  proposed  algorithm  has  been  implemented  on  a  PC  with  an  i7-930  CPU  and  a  GeForce  GTX 
295  video  card.  Here  we  present  the  results  of  evaluating  algorithm  under  various  scenarios.  In  [13], 
it  is  stated  that  the  LDI-based  volume  calculation  “is  restricted  to  image  space  precision.” 
Consequently,  it  is  important  to  evaluate  how  the  algorithm  is  influenced  by  insufficient  volume 
sampling,  which  is  caused  by  low  resolution.  The  generated  haptic  feedback  forces,  based  on 
different  resolutions,  are  corn-pared  with  previously  recorded  high-resolution  (512  x  512)  results. 
The  normalized  absolute  error  is  calculated  as: 

1^-^5121/^12  (?) 

where  is  the  m-resolution  output  force  and  is  the  high-resolution  force  used  as  a  benchmark. 


Three  scenarios  are  adopted  to  test  the  algorithm.  In  each  scenario,  two  objects  are  colliding.  The 
objects  used  in  each  test  environment  are:  (1)  a  tumor  and  a  cylindrically-shaped  object;  (2)  a 
tumor-shaped  object  and  a  duo-denum-shaped  object;  and  (3)  a  tumor-shaped  object  and  a  bunny¬ 
shaped  object.  The  numbers  of  triangles  of  the  aforementioned  meshes  are:  tumor:  10,000;  cylinder: 
1,340;  duodenum:  2,000;  bunny:  70,000.  The  simulation  scenarios  and  the  feedback  force  error  are 
shown  as  below: 


Figure  2.  Tumor-cylinder  test  scenario 
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Figure  5.  Error  comparison  for  tumor-duodenum 
scenario 


Figure  7.  Error  comparison  for  tumor-bunny 
scenario 


From  Figures  3,  5  and  7,  we  can  easily  see  that  volume  errors  do  have  a  negative  impact  on  the 
haptic  force.  However,  in  most  cases,  the  influence  is  less  than  3%.  Therefore,  the  lower 
resolution  setting  can  also  be  used  for  real-time  simulations. 

Another  aspect  critical  in  haptic  simulation  is 
haptic  refreshment  rate.  To  evaluate  the 
computation  time  cost  of  proposed  algorithm,  we 
tested  the  haptic  frame  rate  for  the  three 
intersection  scenarios.  Fig.  8  shows  the  frame  rate 
comparison. 


Judging  from  the  error  and  frame  rate,  we  can  con¬ 
clude  that,  although  adopting  a  resolution  of  32 
can  yield  a  higher  haptic  response  rate,  the  error 
may  be  relatively  large.  Using  a  resolution  of  64  is 
more  reliable  and  yields  an  acceptable  refreshment 
rate  of  approximately  200  FPS.  Since  the  proposed 
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algorithm  implementation  GPU-based,  the  CPU  can 
process  its  own  tasks  in  parallel,  while  the  LDI  is 
being  generated  on  the  video  card. 

We  use  a  scene  describing  a  “cylinder  sweeps  across 
a  cube”  to  demonstrate  that  by  introducing  different 
local  stiffness  vertices,  the  local  stiffness  vertex 
method  can  lead  to  a  varying  feedback  force  effect. 

The  scene  is  shown  in  Fig.  9.  In  this  scenario,  when 
no  local  stiffness  vertices  are  considered,  the  only 
haptic  feedback  force  calculated  is  a  constant  force 
always  facing  the  positive  Y-axis  in  this  scenario. 

To  artificially  introduce  different  stiffness  properties 
at  individual  regions  of  this  scene,  i.e.  making  the 
cube  feel  stiffer  or  softer  at  different  locations,  we 
consider  three  stiffness  vertex  placement  scenarios:  (1)  Conventional  setting,  no  local  stiffness 
vertex  is  included.  (2)  One  local  stiffness  vertex  with  a  large  influence  range  is  located  at  position 
“a”  in  Fig.  9.  In  this  scenario,  the  middle  part  of  cube  will  generate  additional  force,  making  it  feel 
stiffer.  (3)  Two  local  stiffness  vertices  with  different  influence  ranges  and  stiffness  values  at 
locations  “b”  and  “c”  in  Fig.  9.  In  this  scenario,  the  cube  will  feel  stiffer  at  location  “b”  and  “c” 
while  softer  in  the  center.  In  Fig.  10,  the  feedback  forces  in  all  three  scenarios  are  monitored  and 
compared.  Feedback  forces  are  normalized  to  scenario  (1),  the  non-vertex  case  calculated  force.  The 
kernel  function  adopted  is  a  raised  cosine  function: 

a(x )  =  j(l  +  cos(jr))  (8) 

In  Fig.  10,  the  feedback  force  is  measured  as  the  cylinder  travels  from  point  “b”  to  “c”.  Monitored 
forces  show  the  proposed  algorithm  successfully  renders  the  desired  stiffness  properties.  By 
introducing  the  local  stiffness  vertex  method,  the  output  force  is  drastically  converted  from  its 
original  flat  pattern  into  a  curve.  If  one  can  carefully  spread  local  stiffness  vertices  over  the  target 
tissue,  realistic  heterogeneous  stiffness  properties  can  be  easily  perceived,  thus  further  enhancing 
the  realism  of  surgery  simulators. 


We  have  integrated  the  haptic  feedback  algorithm  in 
to  our  simulator  with  constantly  deforming  objects. 
Ver-tices  with  higher  stiffness  values  are  scattered 
around  the  deformable  tissue  representing  the  tumor 
area  in  an  organ.  Surgical  residents  can  use  multiple 
tools  to  touch  and  feel  the  location  of  these  tumors 
for  training  exercises. 


3.  Conclusions 


The  main  contribution  of  this  paper  is  an  LDI-based 
hap-tic  feedback  algorithm  for  deformable  objects 
with  large  volume  intersections.  Unlike  past 
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algorithms  based  on  simplifying  the  haptic  model,  our  method  operates  on  the  original  volume  so 
that  the  irregular  objects  interactions  are  accurately  presented  and  rendered.  Thus,  the  algorithm 
can  be  widely  applied  in  open  surgery  simulations  for  different  instruments  and  arbitrary 
anatomy. 

Future  work  in  this  algorithm  will  include  reinforcing  the  post-processing  method  for  rendering 
different  physical  properties  and  refining  the  parameters  such  as  stiffness  factor,  radius  and  kernel 
function  in  post  processing.  We  will  work  closely  with  our  surgical  collaborators  to  address  these 
issues. 
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Chen,  C.-P.,  Sethi,  D.,  Jin,  Y.-Y.,  and  Wickstrom,  E.  (2011)  EGFR-targeting  peptide  as  ligand 
to  direct  peptide  nucleic  acid  (PNA)  for  imaging  oncogene  mutations.  22nd  American 
Peptide  Symposium,  San  Diego,  California,  June  25-30. 

Design  and  preparation  of  genetic  imaging  agents  for  noninvasive  molecular  imaging  of 
oncogene  expression  in  cancer  cells  is  of  unique  importance  for  cancer  diagnosis  and 
therapy.  EGFR  mutations  occur  in  non-small  cell  lung  cancer,  and  activating  KRAS 
mutations  are  found  in  30%  of  lung  adenocarcinomas.  We  have  synthesized  bioconjugates 
containing  a  receptor  ligand  to  target  the  probes  to  EGFR  overexpressed  on  malignant 
cells,  PNA  to  target  KRAS  mutations,  and  chelator  or  fluorophore  for  imaging. 

Tomographic  imaging  of  positrons  emitted  upon  decay  of  64Cu  will  identify  sites  of  high 
oncogene  expression.  Fluorescently  labeled  PNA-peptide  hybrids  were  prepared  by  solid 
phase  coupling  of  Fmoc-monomers. 
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Sanders,  J.  M.,  and  Wickstrom,  E.  (2011)  Ligand-dependent  activation  and  the  structural 
basis  of  negative  cooperativity  in  EGFR:  a  molecular  dynamics  study.  Protein  Society  25th 
Anniversary  Symposium,  Boston,  Massachusetts,  July  23-27. 

The  Erythroblastic  Leukaemia  Viral  Oncogene  Homolog  (ErbB)  receptor  family  consists 
of  four  structurally  related  receptor  tyrosine  kinase  (RTKs)  involved  in  numerous 
cellular  processes.  Each  member  of  the  ErbB  family  contains  an  extracellular  domain,  a 
single  pass  transmembrane  domain  and  an  intracellular  kinase  domain  regulated  by  a 
juxtamembrane  domain.  Twelve  known  ligands  have  been  identified  that  control  ErbB 
receptor  activation.  In  the  absence  of  an  activating  ligand,  each  ErbB  receptor  adopts  an 
auto-inhibited  conformation,  with  the  exception  of  ErbB2,  controlled  by  several  loose 
intramolecular  interactions.  Ligand  binding  to  the  extracellular  domain  of  an  ErbB 
receptor  promotes  large  scale  conformational  changes  that  allow  for  receptor 
dimerization.  ErbBl,  or  Epidermal  Growth  Factor  Receptor  (EGFR),  was  the  first  to  be 
identified  and  is  the  most  wells  studied  as  aberrant  EGFR  signaling  has  been  identified 
in  multiple  cancer  types.  While  biochemical  and  biophysical  studies  of  the  extracellular 
domain  of  EGFR  (sEGFR)  have  enhanced  our  knowledge  of  ligand-dependent 
activation,  little  is  known  about  the  intrinsic  dynamics  of  the  receptor  on  the  cell 
surface.  Molecular  Dynamics  (MD)  simulations  allow  us  to  explore  dynamics  that  are 
otherwise  inaccessible  with  bench  experiments.  Here  we  demonstrate  that  EGF  binding 
to  sEGFR  alters  the  conformational  sampling  by  disrupting  coupling  of  key  auto- 
inhibitory  interactions.  To  address  the  allosteric  effects  of  EGF  binding,  we  simulated  a 
singly  ligated  sEGFR  dimer.  Our  results  suggest  that  asymmetric  in  the  binding  interface 
caused  by  EGF  binding  distorts  the  second  binding  site.  The  occlusion  of  the  second 
binding  site  most  likely  explains  the  negative  cooperativity  observed  in  cell  binding 
assays  and  argues  against  the  existence  of  distinct  EGFR  receptor  populations.  Our 
results  are  confirmed  by  experimental  x-ray  structures  of  Drosophila  EGFR,  a  homolog 
to  human  EGFR.  These  observations,  previously  not  observed  experimentally,  greatly 
enhance  our  understanding  of  the  molecular  mechanism  of  human  EGFR  activation  and 
may  be  applicable  in  other  ErbB  systems. 
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ABSTRACT 

The  epidermal  growth  factor  receptor  (EGFR)  is  a  member  of  the  ErbB  receptor  tyrosine  kinase 
(RTK)  family  that  plays  a  pivotal  role  in  cell  proliferation,  differentiation  and  migration.  While 
x-ray  structures  have  provided  insight  into  EGFR  function,  the  conformational  pathway(s)  of 
ligand-dependent  activation  are  not  well  understood.  Using  molecular  dynamics  (MD) 
techniques  we  demonstrate  that  EGF  binding  to  the  extracellular  domain  of  EGFR  (sEGFR) 
disrupts  auto-inhibitory  intramolecular  interactions  by  increasing  the  distance  between  the  so- 
called  “dimerization  arm”  and  domain  IV.  To  investigate  early  ligand  binding  effects  on  sEGFR 
dimers,  we  constructed  a  singly  ligated  sEGFR  dimer  for  simulation.  MD  simulations  of  the 
singly  ligated  sEGFR  dimer  exhibited  asymmetrical  motion  in  each  monomer  and  a  decrease  in 
the  size  of  the  second  EGF  binding  pocket  by  10  A.  Free  energy  calculations  indicated  that 
asymmetrical  dimerization  with  only  one  EGF-sEGFR  monomer  is  more  energetically  favorable 
than  dimerization  of  two  ligated  sEGFR  monomers  by  2.88  kcal-mol'1.  To  see  if  asymmetrical 
motion  can  be  extended  to  ErbB  heterodimers,  we  simulated  a  model  sEGFR-ErbB2  dimer.  This 
heterodimer  also  exhibited  asymmetrical  motion.  These  results  suggest  asymmetrical  dynamics 
in  sEGFR  may  play  an  essential  role  in  receptor  activation  and  cooperativity. 


KEYWORDS:  Epidermal  growth  factor,  EGFR,  cooperativity,  ErbB2,  molecular  dynamics, 
protein  dynamics 

INTRODUCTION 

The  ErbB  receptor  family  consists  of  four  structurally  related  receptor  tyrosine  kinases 
(RTK)  that  play  pivotal  roles  in  cell  differentiation,  proliferation  and  migration  [10].  Early 
studies  of  ErbB  receptors  were  the  first  to  highlight  the  importance  of  ligand- induced 
dimerization  in  intracellular  signalling  [11],  EGFR  is  one  of  the  most  well  studied  RTKs  as 
aberrant  signalling  and  mutations  in  EGFR  have  been  implicated  in  several  cancer  types  Several 
crystallographic  and  biophysical  studies  have  shed  light  on  the  mechanism  of  ligand-dependent 
dimerization  leading  to  receptor  activation  [1,  12,  13].  These  studies  demonstrated  that  large 
scale  conformational  changes  occur  in  EGFR  upon  EGF  binding,  shifting  the  conformational 
equilibrium  from  an  auto-inhibited  (tethered)  state  to  an  un-inhibited  (untethered)  dimer- 
competent  state  (Figure  1).  While  the  process  of  EGF-induced  dimerization  of  EGFR  is  well 
characterized,  several  gaps  in  the  molecular  mechanism  of  EGFR  activation  still  remain, 
particularly  the  negative  cooperative  effects  EGFR  exerts  on  EGF  binding  [14-16], 
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The  first  x-ray  structure  of  the  monomeric  extracellular  domain  of  EGFR  (sEGFR) 
showed  4  domains  spanning  620  amino  acids  in  an  auto-inhibited  conformation,  based  on  several 
intramolecular  interactions  between  domains  II  and  IV  [1].  Domains  I  and  III  adopt  a  P-solenoid 
fold  and  are  homologous  ligand  binding  domains,  while  II  and  IV  are  cysteine-rich  domains  with 
extended  rod-like  structures.  Structures  of  the  other  ErbB  receptors  [17-19]  showed  several 
common  features,  with  the  exception  of  ErbB2,  including  a  dimerization  arm  in  domain  II 
interacting  with  domain  IV  and  additional  intramolecular  interactions  that  prevent  dimerization 
in  the  absence  of  an  activating  ligand  (Figure  2).  Results  of  single  molecule  optical  imaging 
studies  of  EGFR  indicated  that  dimerization  can  occur  without  an  activating  ligand  [20]. 
However,  ligand-independent  EGFR  dimer  complexes  were  transient;  kinase  activity  was  not 
detectable  [20],  Small  angle  x-ray  scattering  studies  of  sEGFR  concluded  that  receptor 
dimerization  is  ligand-dependent  as  sEGFR  adopts  a  compact  auto-inhibited  conformation  in  the 
absence  of  EGF  and  upon  addition  of  EGF  forms  a  dimer  in  vitro  [13]. 

While  these  studies  have  shown  that  EGFR  adopts  two  distinct  conformational  states  in 
the  presence  or  absence  of  ligand  binding,  the  dynamics  of  sEGFR  are  still  poorly  understood. 
Previous  molecular  dynamics  (MD)  simulations  of  EGFR  dimers  have  explored  dynamics  under 
different  solvent  conditions,  higher  order  oligomer  states,  and  interactions  with  the  cell 
membrane  surface  [21,  22],  What  have  not  been  investigated  are  the  initial  events  that  occur 
upon  ligand  binding  that  drive  the  conformational  transitions  necessary  to  form  EGFR  dimers. 

EGF  binding  to  EGFR  is  thought  to  be  a  negatively  cooperative  process.  Scatchard  plots 
of  EGF  binding  yield  a  concave  up  curve,  indicative  of  either  multiple  independent  sites  or 
negative  cooperativity  [14-16].  The  existence  of  multiple  independent  sites  has  been  shown  to  be 
unlikely,  as  independent  equivalent  two-site  modeling  of  EGF  binding  curves  does  not  agree 
with  previous  experiments  [23],  In  contrast,  asymmetrical  binding  events  on  an  EGFR  dimer  led 
to  a  model  for  negative  cooperativity  [14].  That  view  was  reinforced  by  a  recently  solved  crystal 
structure  of  a  singly  ligated  EGFR  dimer  from  Drosophila  [24],  In  the  case  of  Drosophila,  EGFR 
binding  of  one  Spitz  (EGF  homolog)  molecule  causes  structural  rearrangements  that  partially 
occlude  the  second  binding  site.  Based  on  the  asymmetric  dimer  structure,  it  was  concluded  that 
partial  occlusion  of  the  second  site  may  be  a  common  feature  of  all  EGFRs,  and  may  have 
functional  relevance  in  development  and  in  the  formation  of  ErbB  heterodimers  [25]. 

In  order  to  explore  ligand-dependent  activation  of  sEGFR  and  conformational  effects  that 
might  contribute  to  asymmetric  and  possibly  anti-cooperative  sEGFR  complexes,  we  carried  out 
MD  simulations.  We  examined  models  of  (a)  auto-inhibited  sEGFR  structures  with  and  without 
EGF,  (b)  a  singly  ligated  sEGFR  dimer,  and  (c)  a  sEGFR-ErbB2  heterodimer.  Our  results 
suggest  that  the  extracellular  domain  of  EGFR  might  play  a  role,  not  previously  observed,  in 
regulation  of  negative  cooperativity,  independent  of  transmembrane  and  extracellular  domains. 
We  also  conclude  that  the  failure  of  previous  in  vitro  experiments  to  observe  anti-cooperative 
bind  in  the  extracellular  domain  of  EGFR  may  be  due  to  the  time  scales  (seconds  to  minutes)  of 
each  technique.  The  asymmetric  dimer  conformations  observed  for  a  singly  ligated  EGFR  dimer 
and  EGFR-ErbB  dimer  are  mostly  like  formed  in  the  microsecond  to  millisecond  time  domains 
and  stabilized  by  the  transmembrane  and  intracellular  domains. 
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MATERIALS  AND  METHODS 

Simulation  Models 

Four  systems  were  simulated  in  this  study.  The  first  two  systems  were  based  on  the  monomeric 
holo-sEGFR  structure  including  bound  EGF  (PDB  code  1NQL).  All  oligosaccharides  were 
removed  from  each  polypeptide.  In  the  apo-EGFR  system,  the  EGF  molecule  was  also  removed. 
The  third  model  was  based  on  the  crystal  structure  of  the  activated  sEGFR  dimer  (PDB  code 
1IVO)  with  re-refinement  to  incorporate  domain  IV  (PDB  code  3NJP).  To  generate  a  singly 
ligated  dimer,  one  of  the  EGF  molecules  was  removed.  The  fourth  system  was  constructed  by 
modeling  an  EGFR-ErbB2  heterodimer  based  on  the  EGFR  dimer  structure.  First  the  structure  of 
ErbB2  (PDB  code  1N8Z)  was  edited  by  removing  the  trastuzumab  Fab  fragment.  Several  loops 
were  not  resolved  in  the  ErbB2  crystal  structure,  so  each  loop  was  modelled  with  SWISS- 
MODEL  (http://swissmodel.expasy.org/).  The  heterodimer  was  constructed  by  aligning  the 
dimerization  arm  of  ErbB2  to  the  dimerization  arm  of  one  EGFR  molecule  in  the  EGFR  dimer 
structure.  All  alignments  were  performed  in  PyMOL  and  all  images  were  produced  in  VMD  and 
PyMOL  [26,  27], 

Molecular  Dynamics  (MD)  Simulations 

The  tleap  module  of  AMBER  10  was  used  to  create  parameter  and  topology  files  for  the  MD 
simulations  using  the  AMBER  ffD3  force  field  [28],  Hydrogen  atoms  were  added  and  ionizable 
residues  (histidines)  were  set  to  predicted  protonation  states  at  pH  7.0.  Na+  counterions  were 
added  to  each  system  to  achieve  neutrality.  Solvent  molecules  were  added  using  a  TIP3PBOX 
water  box  with  a  minimum  spacing  of  10.0  A  from  the  box  edges  to  the  protein  molecule.  Each 
system  contained  >200K  atoms. 

Energy  minimization  on  each  system  was  performed  in  a  two-step  process.  First  the  protein 
atoms  were  restrained  and  the  water  molecules  were  allowed  to  relax  over  1000  steps.  The 
entire  system  was  then  subjected  to  energy  minimization  using  the  steepest  descent  method  for 
the  first  1000  steps,  followed  by  the  full  conjugate  gradient  method  for  an  additional  2400  steps. 
Each  system  was  then  heated  to  300K  for  100  ps  followed  by  a  50  ps  constant  pressure 
simulation  to  adjust  the  water  density  to  1  g/mL.  An  additional  500  ps  simulation  was  run  prior 
to  production  simulations  to  allow  for  further  temperature  and  pressure  equilibration.  Production 
runs  were  performed  using  a  canonical  ensemble  (NVT)  scheme.  Langevin  dynamics  with  a 
collision  frequency  of  2.0  were  used  for  temperature  regulation,  ii)  the  SHAKE  algorithm  was 
used  to  constrain  all  covalent  bonds  involving  hydrogen  atoms,  and  iii)  the  particle  mesh  Ewald 
(PME)  method  was  employed  to  treat  long  range  electrostatics  and  van  der  Waals  forces  with  an 
integration  step  of  2.0  fs.  Production  simulations  for  both  EGFR  systems  were  run  for  70  ns.  The 
singly  ligated  EGFR  dimer  was  run  for  25  ns  and  the  EGFR-ErbB2  heterodimer  simulations 
were  performed  for  16  ns. 

Principal  Component  Analysis  (PCA)  of  Trajectories 

MD  trajectories  were  analyzed  and  compressed  using  the  AMBER  module  ptraj  for  analysis. 
PCA  analysis  was  done  using  ProDy  0.6.1  and  MDanalysis  [29,  30].  PCA  was  performed  by 
decomposing  the  covariance  matrix  for  each  snapshot  of  the  trajectory  ensemble.  The  covariance 
matrix  is  a  3N  x  3N  matrix  for  a  protein  of  N  residues  given  by  their  C-a  atom  coordinates  (x)  in 
phase  space  (i,j  =  1,...,3N)  and  is  computed  as 
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<7  =  <(Xi-<X>i)(Xj-<Xj>)>. 

The  eigenvectors,  also  called  the  principal  components  (PCs),  correspond  to  a  linear  basis  set  of 
the  distribution  of  MD  snapshots.  Projections  of  the  first  three  PCs  were  used  to  construct 
correlated  fluctuation  maps  for  all  residues.  All  PCA  graphs  were  made  using  ProDy  and 
matplotlib  in  python  (http://matplotlib.sourceforge.net/).  All  structural  renderings  of  PCs  were 
made  using  the  Nmwiz  plugin  feature  of  ProDy  in  VMD. 

RMSD  of  trajectory  analysis 

The  RMSD  over  the  time  course  of  each  trajectory  was  calculated  by  using  the  RMSD 
Trajectory  tool  in  VMD.  To  quantify  domain  fluctuations,  the  RMSD  values  for  each  domain  in 
EGFR  were  calculated  for  each  snapshot  in  the  simulation. 

Calculation  of pairwise  distances  and  RMSF  of  residues 

The  RMSF  of  each  residue  were  calculated  using  ptraj  by  processing  each  compressed  MD 
trajectory  for  a  given  system.  Pairwise  distances  between  residues  were  measured  by  calculating 
the  distance  between  C-a  of  each  residue  for  each  snapshot  along  the  simulation  trajectory. 
Pairwise  distances  were  also  calculated  between  the  center  of  mass  for  the  dimerization  arm 
(residues  246-250)  and  the  center  of  mass  for  the  second  disulfide  loop  of  domain  IV  (residues 
558-567). 

Free  Energy  Calculations 

Free  energy  calculations  of  each  simulation  were  performed  using  SEQMOF  (provided  by 
BiochemFabSolution.com)  by  using  coordinate  files  constructed  at  the  last  time  step  of  each 
trajectory.  For  comparison,  crystal  structures  of  singly  ligated  Drosophila  EGFR  (PDB  code 
3FTG),  fully  ligated  Drosophila  EGFR  (PDB  code  3ETF)  and  fully  ligated  human  EGFR  dimer 
(PDB  code  3NJP)  were  also  submitted  to  obtain  theoretical  free  energy  changes. 


RESULTS 

Simulations  of  auto-inhibited  sEGFR 

To  gain  insight  into  the  molecular  mechanism  of  ligand-dependent  activation  of  EGFR, 
we  simulated  four  systems  of  the  extracellular  domain  of  EGFR  (sEGFR)  in  both  auto-inhibited 
and  un-inhibited  conformational  states.  EGFR  is  known  to  exist  in  conformational  equilibrium 
between  these  two  states  (12).  The  addition  of  EGF,  or  other  activating  ligand,  shifts  the 
equilibrium  toward  the  active  state.  To  model  the  early  conformational  changes  induced  by  EGF 
binding  we  performed  simulations  of  the  monomeric  sEGFR  in  the  presence  and  absence  of 
EGF.  After  70  ns  of  simulation  we  analyzed  each  trajectory.  Since  sEGFR  is  a  large  multi- 
domain  protein,  we  did  not  expect  to  capture  the  conformational  pathway  that  leads  to  a  dimer- 
competent  conformational  state  within  the  time  scales  of  conventional  MD  simulations.  Instead 
we  focused  on  conformational  changes  that  occur  immediately  after  EGF  binds  to  domain  I.  We 
noticed  that  the  binding  of  EGF  to  sEGFR  slightly  altered  domain  motion  (Figure  3  A  &  3B). 
EGF  binding  reduced  the  RMSD  fluctuations  in  domain  II  from  5  A  to  4  A.  After  70  ns  the 
sEGFR  without  EGF  (Figure  3D)  adopted  a  similar  conformation  to  the  auto-inhibited  structures 
of  ErbB3  and  ErbB4,  while  the  sEGFR  with  EGF  adopts  a  different ,  possibly  an  intermediate, 
conformational  state. 
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Previous  studies  have  shown  that  intramolecular  interactions  between  the  “dimerization 
arm”  of  domain  II  and  the  “tethering  arm”  of  domain  IV  are  important  for  preventing  ligand- 
independent  activation  and  negative  cooperativity  by  dimerization  (Figure  2A)  [31].  To  explore 
this  role  of  the  dimerization  arm,  we  monitored  the  trajectories  of  the  dimerization  arm  and 
domain  IV.  Our  prediction  was  that  EGF  binding  should  force  the  dimerization  arm  to  disengage 
from  domain  IV.  To  quantify  our  results  we  measured  the  distance  between  the  center  of  mass  of 
the  tip  of  the  dimerization  arm  (residues  246-250)  and  the  center  of  mass  of  the  second  disulfide 
loop  (residues  558-567)  on  domain  IV  (Figure  5  A).  In  the  absence  of  EGF,  the  change  in 
distance  between  the  two  loops  was  less  than  1  A.  EGF  binding,  however,  caused  a  stepwise 
increase  in  separation  distance  during  the  70  ns  course  of  the  simulation  (Figure  5B). 

To  determine  the  “essential  dynamics”  of  the  dimerization  arm  and  we  performed  PC  A 
on  the  MD  trajectories  for  sEGFR  and  EGF  bound  to  sEGFR.  PCA  is  frequently  used  to 
determine  the  direction  and  greatest  variance  in  atomic  motion  in  MD  simulations  and  has  also 
been  used  to  predict  conformational  changes  from  multiple  crystal  structures  [32-34],  PCA  of 
each  simulation  showed  altered  conformational  sampling  of  the  dimerization  arm  (Figure  4).  In 
the  absence  of  EGF,  both  the  dimerization  arm  and  tethering  arm  fluctuated  about  an  equilibrium 
position  with  no  net  correlated  motion.  Binding  of  EGF  to  domain  I  resulted  in  increased 
fluctuations  of  the  dimerization  arm  and  an  increase  in  correlated  motion  between  the 
dimerization  arm  and  the  tethering  arm. 

Simulations  of  singly  ligated  sEGFR  dimer 

To  investigate  the  negative  cooperative  effects  observed  in  binding  assays  (23),  we 
performed  a  simulation  with  a  singly  ligated  sEGFR  dimer.  Recently  the  structures  of 
Drosophila  EGFR  (dEGFR)  in  different  ligation  states  were  solved  [24],  Drosophila  EGFR  is 
highly  homologous  (35%)  to  human  EGFR  and  retains  negative  cooperativity  in  vitro,  unlike 
human  sEGFR.  Drosophila  EGFR  forms  an  asymmetric  dimer  that  is  loosely  associated  when 
one  Spitz  ligand  and  exhibits  half-site  reactivity.  These  effects  have  not  been  seen  previously  in 
biophysical  studies  for  human  sEGFR.  We  therefore  chose  to  simulate  a  singly  ligated  human 
sEGFR  with  the  prediction  that  it  would  adopt  an  asymmetrical  conformation  similar  to  that  seen 
in  Drosophila  EGFR.  Results  from  the  25  ns  simulation  confirmed  that  dimerization  was 
retained  in  the  presence  of  only  one  EGF  molecule,  but  imposed  structural  restraints  on  the 
unligated  portion  of  the  sEGFR  dimer,  resulting  in  an  asymmetrical  conformation. 

We  used  PCA  to  analyze  the  asymmetrical  motion  observed  in  the  singly  ligated  dimer 
simulation.  Motion  along  the  first  PC  in  the  unligated  half  was  greatly  affected  by  the  lack  of 
EGF  bound  to  domains  I  and  III.  The  motion  in  domain  II  along  the  dimer  interface  was 
asymmetrical  but  did  break  contacts  made  by  each  dimerization  arm.  These  motions  were 
propagated  to  domains  I  and  III  on  the  unligated  half.  The  positions  of  these  domains  were  held 
fixed  in  the  EGF-ligated  half.  Without  EGF  in  the  unligated  portion,  domains  I  and  III  moved 
toward  each  other,  distorting  the  binding  pocket  and  decreasing  the  accessible  surface  area  for  a 
ligand  to  bind. 

To  quantify  the  motions  of  domains  I  and  III,  we  measured  the  distances  between  three 
pairs  of  residues  on  different  sections  of  the  EGF  binding  surface:  dl  (Tyr45  to  Phe357),  d2 
(Asnl2  to  His346)  and  d3  (Arg29  to  Phe412).  By  the  end  of  a  25  ns  simulation,  all  three 
distances  decreased  on  the  unligated  half  relative  to  the  ligated  half  (Figure  6).  The  greatest 
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change,  10  A,  occurred  in  the  distance  between  Tyr45  and  Phe357  by  for  the  unoccupied  binding 
site,  while  the  EGF -bound  half  remained  constant. 

In  addition  to  monitoring  structural  changes,  we  sought  to  determine  the  energetics  of 
binding  for  a  second  EGF  to  a  singly  ligated  dimer.  If  sEGFR  retains  negative  cooperativity, 
there  should  be  a  difference  in  binding  affinities  between  each  binding  site  on  the  dimer.  Using 
SEQMOF’s  free  energy  solver,  we  calculated  the  binding  energy  for  EGF  binding  after  25  ns  of 
simulation  (Table  1).  We  obtained  a  free  energy  change  of  AG  =  -  23.16  kcal-mol'1  for  EGF 
binding  at  298  K  (25°C).  The  free  energy  of  dimerization  was  also  calculated  and  found  to  be 
AG  =  -20.66  kcal-mol'1. 

To  compare  our  results  to  a  fully  ligated  sEGFR  dimer  we  performed  the  same  analysis 
on  the  sEGFR  dimer  structure  with  domain  IV  modelled.  Unexpectedly,  the  binding  energies  for 
each  EGF  molecule  were  not  equal  prior  to  relaxing  the  structure  using  MD.  The  difference 
between  binding  events  was  1.96  kcal-mol'1,  which  might  reflect  asymmetry  within  the  crystal 
structure.  Differences  in  dimerization  energy  are  another  possible  factor  that  might  lead  to 
negative  cooperativity.  A  single  sEGFR  with  bound  EGF,  dimerizing  with  a  second  unligated 
sEGFR,  might  be  more  energetically  favorable  than  dimerization  of  two  EGF+sEGFR 
molecules.  The  AAG  for  the  difference  was  -2.78  kcal-mol'1,  suggesting  that  asymmetrical 
dimerization  is  more  energetically  favorable  than  symmetrical  dimerization  with  both  sEGFR 
bound  to  EGF. 

We  compared  these  results  with  free  energy  calculations  on  the  two  asymmetrical 
Drosophila  EGFR  crystal  structures  that  exist  in  singly  ligated  and  fully  ligated  states.  A  fully 
ligated  Drosophila  EGFR,  unlike  sEGFR,  appears  as  an  asymmetrical  dimer  in  the  crystal 
structure.  As  in  the  sEGFR  dimer  crystal  structure,  the  binding  energies  of  each  Spitz  molecule, 
a  Drosophila  EGF  homolog,  were  not  equal  (AAG  =  1.90  kcal-mol'1).  The  free  energy  change  of 
dimerization,  however,  was  more  negative  than  the  value  for  sEGFR+EGF  dimerization  (AAG  = 
-2.53  kcal-mol'1).  This  agrees  with  our  results  from  the  singly  ligated  sEGFR  dimer  MD 
simulation.  For  the  case  of  singly  ligated  Drosophila  EGFR  bound  to  a  low  affinity  mutant  Spitz, 
the  dimerization  energy  is  less  favorable  (AAG  =  -13.96  kcal-mol"1).  This  effect  might  be  a  result 
from  structural  restraints  imposed  by  the  Spitz  mutant,  although  the  binding  energy  for  ligand  in 
this  case  was  -29.06  kcal-mol'1,  the  lowest  energy  ligand-binding  event  in  this  study.  It  is  worth 
noting  that  there  is  less  buried  surface  by  the  mutant  (AASA  =  -3146.6  A2)  than  both  wild-type 
Spitz  molecules  (AASA  =  -3796.1  A2  &  -41 16.2  A2). 

Simulation  of  heterodimerization  in  EGFR  and  ErbB2 

To  date  no  structure  exists  of  an  ErbB  heterodimer  and  little  is  known  about  the 
molecular  mechanism  of  heterodimerization  of  ErbB  receptors.  We  constructed  a  model  of  an 
EGF  bound  sEGFR-ErbB  heterodimer  for  MD  simulation.  This  was  done  by  aligning  the 
structure  of  ErbB2  to  one  half  of  the  symmetric  human  sEGFR  dimer  using  the  dimerization  arm 
as  a  reference.  This  heterodimer  was  then  subjected  to  a  16ns  simulation.  After  16  ns  the 
sEGFR-ErbB2  dimer  adopted  an  asymmetrical  dimer  (Figure  7).  Unlike  the  singly  ligated 
sEGFR  dimer,  the  heterodimer  buried  less  surface  at  the  dimer  interface  (AASA  =  -3581.7  A2). 
The  energy  of  dimerization  for  a  sEGFR+EGF -ErbB2  heterodimer  was  lower  than  the 
dimerization  of  a  doubly-  ligated  sEGFR  (AAG  =  -2.02  kcal-mol'1).  EGF  binding  to  a 
heterodimer  was  only  slightly  more  favorable,  less  than  1  kcal-mol'1,  according  to  predicted 
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binding  energies.  This  suggests  that  binding  of  EGF  to  sEGFR  is  not  dependent  on  the  other 
subunit,  provided  the  other  subunit  not  already  occupied  by  a  ligand. 

DISCUSSION 


EGFR  is  a  member  of  the  four  structurally  related  receptors  that  are  necessary  for  cellular 
development  and  homeostasis.  The  extracellular  domain  of  EGFR  (sEGFR),  in  the  absence  of 
an  activating  ligand,  adopts  an  auto-inhibited  “tethered”  state.  Upon  binding  of  an  active  ligand 
sEGFR  undergoes  conformational  rearrangements  that  allow  it  to  dimerize  with  itself  or  another 
ErbB  receptor.  sEGFR  is  a  highly  dynamic  protein  that  is  thought  to  sample  both  the  inhibited 
state  and  less  frequently  the  uninhibited  state  without  ligand  binding.  To  explore  the  structural 
changes  that  occur  upon  EGF  binding  we  performed  70ns  MD  simulations  on  monomeric 
sEGFR  in  the  presence  and  absence  of  EGF. 

The  dimerization  arm  in  domain  II  is  thought  to  be  involved  in  restraining  the  auto- 
inhibited  state.  Hence,  EGF  binding  might  alter  the  conformational  sampling  of  the  arm  to  break 
the  intramolecular  interactions  observed  in  the  crystal  structure.  Using  pairwise  distance 
measurements  between  the  dimerization  arm  and  the  tethering  arm  of  domain  IV,  we  observed 
an  EGF-dependent  increase  in  separation  over  the  course  of  a  70  ns  simulation.  While  we 
noticed  fluctuations  in  the  absence  of  EGF,  the  distance  at  the  end  of  the  simulation  had  not 
changed  relative  to  the  starting  structure.  This  suggests  that  EGF  may  be  interacting  with  domain 
II  while  being  bound  to  domain  I,  effectively  pulling  the  dimerization  arm  away  from  the  tether 
arm  of  domain  IV.  Our  PCA  results  revealed  that  the  correlated  motion  between  the  two  arms 
increased  when  EGF  is  bound.  While  we  did  not  observe  a  complete  disengagement  of  the 
dimerization  arm  in  our  simulations,  the  increased  distance  of  the  dimerization  arm  relative  to 
domain  IV  agrees  with  the  model  that  EGF  disrupts  the  auto-inhibited  conformation,  exposing 
the  arm  for  dimerization.  The  lack  of  correlated  motion  and  no  net  increase  in  distance  in  the 
absence  of  EGF  also  agree  with  previous  results  of  small  angle  x-ray  scattering  studies  of 
sEGFR  [13]. 

Several  reports  have  suggested  that  existence  of  high  affinity  and  low  affinity  EGFR 
populations  on  a  cell  surface  are  a  result  of  negative  cooperativity.  From  these  studies  it  was 
proposed  that  structural  rearrangements  in  an  EGFR  dimer  might  give  rise  to  nonequivalent 
binding  sites.  The  structure  of  a  homologous  EGFR  from  Drosophila  revealed  an  asymmetrical 
dimer  where  the  binding  one  of  ligand  distorted  the  unoccupied  binding  pocket.  This  may  be  the 
case  for  human  sEGFR,  despite  the  apparent  symmetry  observed  in  the  2:2  EGF/sEGFR  crystal 
structure.  Using  MD  we  simulated  a  1 :2  sEGFR  dimer  with  the  prediction  that  it  would  adopt  an 
asymmetrical  conformation  similar  to  the  Drosophila  EGFR.  The  results  of  a  25  ns  simulation 
showed  that  a  singly  ligated  sEGFR  adopted  a  different  structure  from  the  crystal  structure,  in 
agreement  with  the  prediction. 

Using  PCA,  we  found  asymmetrical  motion  along  the  dimer  interface  in  domain  II.  The 
allosteric  effect  might  be  transferred  by  interactions  at  the  interface  and  transmitted  to  the  EGF 
binding  domains  I  and  III.  This  allosteric  signal  transmission  in  protein  dimers  along  a  dimer 
interface  has  also  been  observed  in  MD  studies  of  NikR  and  the  N-terminal  domain  of  the 
AMPA  receptor  [35,  36].  In  the  case  of  a  singly  ligated  sEGFR  MD  simulation,  asymmetrical 
motion  appeared  to  force  domains  I  and  III  to  close  in  a  clamshell  manner.  This  behavior  has 
also  been  observed  in  ligated/unligated  structures  of  the  Spitz  binding  protein  Argos  [37],  In  the 
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absence  of  ligand,  the  ligand  binding  domains  of  Argos  collapse  into  each  other.  This  effect  was 
seen  with  domains  I  and  III  in  the  unligated  binding  pocket  at  25  ns.  Using  distance 
measurements  of  residues  between  the  two  domains,  we  observed  a  decrease  in  all  3 
measurements.  The  most  dramatic  decrease  occurred  between  Y45  and  F357.  In  the  crystal 
structure  this  vector  goes  through  the  core  of  the  EGF  molecule  in  between  the  anti-parallel  beta 
sheets.  This  suggests  that  in  order  for  the  second  EGF  to  bind,  it  must  pry  apart  domains  I  and 


III. 


If  there  is  a  structural  effect  that  decreases  binding  affinity,  then  the  binding  energy 
should  also  be  lowered.  Using  our  25  ns  simulation  data,  we  calculated  binding  energies  for  the 
singly  ligated  sEGFR  dimer  and  compared  them  to  predicted  binding  energies  for  the  crystal 
structures  of  the  2:2  EGF:sEGFR  dimer  and  the  two  Drosophila  EGFR  dimers.  We  found  that  it 
was  more  energetically  favorable  for  one  EGF,  or  Spitz,  to  bind  to  an  asymmetrical  sEGFR,  or 
Drosophila  EGFR  dimer,  respectively,  than  to  a  symmetrical  dimer.  For  the  case  of  a  2:2 
Spitz:  sEGFR  dimer,  the  binding  energy  of  the  first  Spitz  was  lower  than  that  of  the  second  Spitz. 
This  reflects  a  difference  of  binding  affinity,  albeit  small,  that  may  be  correlated  with 
asymmetrical  conformations.  A  recent  fluorescence  resonance  energy  transfer  study  agrees  with 
this  interpretation  [38]. 

In  addition  to  ligand  binding,  we  were  also  interested  in  determining  the  energies  of 
symmetric  and  asymmetric  dimerization.  If  dimerization  of  a  ligated  sEGFR  and  an  unligated 
untethered  sEGFR  is  more  favorable  than  two  ligated  monomers,  this  should  be  reflected  in 
differences  in  binding  energy.  sEGFR  dimerization,  according  to  our  MD  calculations,  was  more 
favorable  when  one  of  the  sEGFR  monomers  was  bound  to  EGF.  A  similar  conclusion  was 
reached  independently,  based  on  mutational  analysis  that  altered  the  cooperativity  of  sEGFR 
(23).  Artificial  sEGFR  heterodimer  experiments  may  be  necessary  to  experimentally  determine 
the  structural  rearrangements  associated  with  singly  ligated  sEGFR  formation. 

While  several  structures  exist  for  EGFR  homodimers,  no  crystal  structure  of  an  EGF 
heterodimer  has  been  solved  at  this  time.  Since  heterodimerization  is  a  common  feature  of  ErbB 
receptor  function,  understanding  the  molecular  details  would  not  only  provide  us  with  a  better 
picture  of  how  ErbB  receptor  activation  occurs,  but  also  a  possible  role  of  the  extracellular 
domain  in  combinatorial  complexity  in  downstream  biological  signals.  Our  MD  simulation 
provides  the  first  picture  of  a  possible  sEGFR-ErbB2  heterodimer  structure.  Using  theoretical 
free  energy  calculations,  we  observed  little  energy  difference  between  homodimerization  and 
sEGFR-ErbB2  heterodimerization.  A  smaller  surface  area  was  buried  between  the  two  proteins, 
which  might  alter  the  intracellular  kinase  domains  ability  to  recruit  scaffolding  proteins.  Since 
ErbB  overexpression  is  associated  with  several  carcinomas,  this  model  of  sEGFR-ErbB2 
heterodimerization  could  lead  to  the  development  of  peptide  or  antibody  inhibitors  that 
specifically  target  heterodimers  or  block  dimerization. 

CONCLUSIONS 

In  this  study  we  sought  to  map  the  conformational  pathway  of  ligand  dependent 
activation  in  the  extracellular  domain  of  EGFR.  Using  molecular  dynamics  simulations  we 
observe  changes  in  domain  fluctuations  upon  binding  of  EGF  to  EGFR  in  the  auto-inhibited 
state,  particularly  between  the  dimerization  arm  and  domain  IV.  Since  exposure  of  the 
dimerization  arm  is  necessary  for  EGFR  dimerization,  the  trajectory  of  the  sEGFR  with  EGF 
may  represent  an  intermediate  state  before  the  active  conformer. 
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Several  studies  have  suggested  the  possibility  that  asymmetrical  dimerization  may  be  a 
common  feature  of  RTKs.  To  explore  this  hypothesis  we  generated  two  asymmetric  dimers,  one 
for  a  singly  ligated  sEGFR  and  EGFR-ErbB2,  and  performed  explicit  solvent  calculations.  PCA 
and  binding  energy  calculation  results  suggest  sEGFR  dimerization  is  more  favorable  for  one 
EGF  occupied  monomer  and  one  unoccupied.  These  asymmetrical  dynamics  in  sEGFR  might 
play  a  role  in  the  negative  cooperative  effects,  in  agreement  with  previous  in-vitro  binding 
experiments  (23).  Such  structural  transitions  might  have  a  functional  role  in  promoting  cell 
signalling  with  ErbB  heterodimerization  as  seen  in  our  EGFR-ErbB2  heterodimer. 

Using  molecular  dynamics  simulations  to  generate  stable  conformational  states  otherwise 
inaccessible  could  be  used  to  rationally  design  conformation-specific  antibodies  for  ErbB- 
targeted  therapeutics.  These  results  provide  a  glimpse  at  the  conformational  landscape  of  EGFR 
and  can  serve  as  a  template  for  enhanced  conformational  sampling  techniques  and  full  length 
receptor  models. 
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Table  1.  Predicted  Binding  Energies  at  298  K  (25°C). 


Reaction 

^Esoiy 

AEeiec 

AH 

(kcal/mol) 

AS 

(kcal/mol-K) 

AASA  (A2) 

AG 

(kcal/mol) 

EGF-sEGFR  +  EGF-sEGFR  <->  EGF2-sEGFR2 
(crystal  structure) 

-29949.6 

117.71 

21 

276 

-3463.8 

-17.88 

EGF1  +  EGF2-sEGFR2  <->  EGF2-sEGFR2 
(crystal  structure) 

-23367.4 

-258 

23 

257 

-3104.1 

-19.59 

EGF2  +  EGF‘-sEGFR2<->  EGF2-sEGFR, 

(crystal  structure) 

-25060 

-229.45 

22 

273 

-3154.2 

-17.63 

dEGFR-Spitzmutant+dEGFR  <->  dEGFR2- 

SpitZmutant 

(crystal  structure) 

-21754.3 

121.64 

28 

357 

-3494.4 

-13.96 

dEGFR2  +  Spitzmutant  <->  dEGFR2-Spitzmutant 
(crystal  structure) 

-30125.3 

-447.84 

23 

174 

-3146.6 

-29.06 

Spitz1  +  Spitz2-dEGFR2  <->  Spitz2-dEGFR2 
(crystal  structure) 

-38231.5 

-288.12 

28 

280 

-3796.1 

-24.17 

Spitz2  +  Spitz1-dEGFR2  <->  Spitz2-dEGFR2 
(crystal  structure) 

-28612.5 

-429.86 

33 

312 

-4116.2 

-22.37 

Spitz  1-dEGFR2  +  Spitz2-dEGFR2  <->  Spitz2- 
dEGFR2 

(crystal  structure) 

-23415 

97.2 

29 

290 

-3536.1 

-20.41 

EGF-sEGFR  +  sEGFR  <->  EGF-sEGFR2 
(25  ns  MD  simulation) 

-35491.4 

-80.1 

33 

349 

-4684.3 

-20.6 

EGF  +  sEGFR2  <->  EGF-sEGFR2 
(25  ns  MD  simulation) 

-12029.4 

-233.85 

26 

228 

-2979.5 

-23.16 

EGF-sEGFR  +  ErbB2  <->  EGF -sEGFR-ErbB2 
(16  ns  MD  simulation) 

-42388.2 

-56.36 

22 

293 

-3581.7 

-19.9 

EGF  +  sEGFR-ErbB2  <->  EGF -sEGFR-ErbB2 
(16  ns  MD  simulation) 

-17163 

-189.7 

29 

242 

-3041.5 

-24.37 
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Figure  Legends 

Figure  1.  Structural  representation  of  the  mechanism  of  ligand-dependent  activation  and 
dimerization  of  EGFR.  The  extracellular  domain  consists  of  two  beta-solenoid  domains  (red)  that 
are  responsible  for  EGF  binding  and  two  cysteine -rich  domains.  The  first  cysteine -rich  domain 
(green)  contains  the  dimerization  arm  necessary  for  dimerization  and  the  second  cysteine-rich 
domain  (gray)  is  involved  in  coordinating  the  dimerization  arm  in  the  absence  of  ligand. 

Figure  2.  sEGFR  intramolecular  and  intermolecular  interactions  that  are  involved  in  receptor 
autoinhibition  and  EGF  binding.  The  dimerization  arm  of  domain  II  (A)  is  capable  of  forming 
several  noncovalent  interactions  with  domain  IV  to  prevent  formation  of  stable  dimers  and 
restrict  conformational  sampling  of  sEGFR.  The  binding  pocket  of  sEGFR  (B)  coordinates  EGF 
(blue)  with  both  domains  I  and  III.  Several  residues  are  responsible  for  ligand  coordination. 

Their  distances  are  symmetric  in  the  sEGFR  dimer  crystal  structure. 

Figure  3.  Domain  motions  in  sEGFR  in  the  presence  and  absence  of  EGF.  RMSD  plots  over  the 
course  of  70  ns  of  simulation  for  sEGFR+EGF  (A)  and  sEGFR  (B).  Structural  representation  at 
the  end  of  70  ns  for  sEGFR+EGF  (C)  and  sEGFR  (D). 

Figure  4.  Binding  of  EGF  alters  the  dynamics  of  the  dimerization  arm  and  increases  coupling  of 
motion  between  the  dimerization  and  tethering  arms.  Correlated  fluctuations  of  the  C-a  carbons 
of  sEGFR+EGF  (A)  and  sEGFR  (B)  show  an  increase  of  coupled  motion  between  domains  II 
and  IV  (black  box)  when  EGF  is  bound  to  domain  I.  The  color  bars  in  A  and  B  indicate  the 
degree  of  coupling,  determined  by  the  normalized  correlation  coefficient.  The  dimerization  arm 
became  more  dynamic  after  ligand  binding  (C)  relative  to  unbound  sEGFR  (D).  Colors  in  C  and 
D  represent  the  positions  of  the  two  arms  during  the  course  of  a  70  ns  simulation.  Blue  traces 
represent  early  positions  in  the  simulation  while  red  traces  indicate  positions  at  the  end  of  the 
simulation. 

Figure  5.  A)  Separation  distance  between  the  dimerization  arm  and  the  S558-S567  loop  of  the 
tethering  arm  of  domain  IV.  B)  Distance  measurements  of  sEGFR  (blue)  and  sEGFR+EGF  (red) 
over  a  70  ns  simulation. 

Figure  6.  Asymmetrical  motion  in  a  singly  ligated  sEGFR  dimer  leads  to  distortion  in  the 
unoccupied  binding  site.  Distance  measurements  were  made  for  three  residue  pairs  between 
domains  I  and  III  in  the  EGF  binding  pocket:  Y101  to  F357  (A),  N12  to  H346  (B)  and  R29  to 
F412  (C). 

Figure  7.  sEGFR  can  form  asymmetrical  homodimers  and  heterodimers.  Structural 
representation  of  singly  ligated  sEGFR  dimer  after  25  ns  of  simulation  (A)  and  an  sEGFR-ErbB2 
heterodimer  (B)  after  16  ns  of  simulation.  Asymmetrical  motion  from  the  results  of  PC  A  for 
singly  ligated  sEGFR  dimer  (C)  and  EGFR-ErbB2  heterodimer  (D).  Black  arrows  indicate  the 
direction  of  motion  for  the  Ca  carbons  along  the  first  principal  component. 
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Abstract  (150-250  words) 

Purpose:  Determine  whether  three-dimensional  (3D)  presentation  of  PET/CT  data  would  benefit 
surgeons  preparing  for  a  procedure. 

Methods:  3D  rendering  and  co-registration  of  PET  and  CT  thorax  scans  using  Amira  5  from  a 
patient  with  a  ductal  pancreatic  mass.  A  group  of  6  surgeons  voluntarily  evaluated  the  usefulness 
of  each  version  of  the  presentation  by  answering  a  questionnaire. 

Results:  Contrast-enhanced  CT  fused  with  PET  provided  a  more  lifelike  presentation  than 
standard  CT,  increasing  the  usefulness  of  the  presentation,  according  to  the  surgeon  evaluations. 
Conclusions:  Volumetric  simulation  of  a  specific  patient,  including  both  CT  and  PET  data,  could 
be  a  useful  practice  tool  for  surgeons  prior  to  making  the  first  incision. 

Introduction 

Surgical  evaluation  of  a  potential  pancreatectomy  is  a  complicated  task  requiring  3D 
spatial  awareness  for  many  structures  of  interest  (tumor,  vasculature,  pancreas,  and  spleen)  and 
how  the  tumor  impacts  upon  each.  This  assessment  will  determine  whether  or  not  a  resection 
should  be  attempted.  If  the  lesion  is  operable,  the  orientation  of  the  pancreas  and  its  surrounding 
structures  will  guide  planning  a  detailed  strategy  for  the  surgical  intervention.  Typically  the 
decision  and  planning  steps  are  based  upon  two  dimensional  (2D)  medical  images  of  the  patient 
from  a  variety  of  sources  including  computed  tomography  (CT),  magnetic  resonance  imaging, 
ultrasound,  photoacoustic  imaging,  positron  emission  tomography  (PET),  and  other  optical 
imaging  using  fluorescence[39].  Planning  for  this  delicate  operation  requires  surgeons  to 
translate  the  2D  image  slices  into  a  3D  mental  picture  of  the  patient’s  anatomy,  which  can  be  a 
difficult  task. 

CT  images  are  a  widely  used,  non-invasive  method  to  study  internal  patient  structures. 
The  2D  images  produced  are  displayed  in  slices  along  the  sagittal,  axial,  and  coronal  planes  of 
the  subject.  Multiple  image  slices  are  taken  of  the  patient  and  put  together  in  their  corresponding 
directions  to  form  a  completed  image  stack.  The  flat  CT  images  are  studied  by  surgeons  to  locate 
diseased  structures  and  study  their  impact  on  surrounding  tissues.  This  can  be  a  difficult  feat,  as 
internal  anatomy  is  a  3D  problem.  The  surgeon  must  pan  through  numerous  2D  images  to 
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generate  a  mental  3D  image  of  important  features,  such  as  the  direction  of  blood  vessels  or  how 
organs  are  positioned  around  each  other. 

With  the  inclusion  of  a  positron-emitting  radiolabel,  specific  features  of  interest  can  be 
observed  by  PET  imaging.  The  specificity  of  the  radiolabel  is  important  for  accurately  marking  a 
target  tissue  for  imaging.  In  particular,  PET  imaging  with  2’-[  FJfluorodeoxy glucose  (  FDG 
PET)  has  become  a  useful  tool  in  the  evaluation  of  tumor  lesions,  particularly  for  pancreatic 
cancer,  and  the  discovery  of  distant  metastases  [40-43],  18FDG  is  a  glucose  analog  that  is  taken 
up  aggressively  by  rapidly  growing  malignant  masses,  but  not  all  hot  spots  are  cancerous. 
Inflamed  or  infected  tissues  also  accumulate  FDG.  The  kidneys  filter  out  excess  sugar  from  the 
blood,  which  results  in  strong  FDG  images.  Likewise  the  bladder  also  shows  a  strong  PET 
image  due  to  accumulation  of  18FDG  waiting  to  be  eliminated. 

CT  and  PET  are  two  examples  of  imaging  methods  used  to  diagnose  and  monitor  cancer 
patients.  Each  has  its  own  individual  advantages  and  drawbacks.  Most  imaging  techniques 
produce  a  series  of  image  slices.  Combining  these  2D  images  into  a  3D  volume  has  the  potential 
to  aid  surgeons  as  they  evaluate  patients  and  prepare  for  surgery.  Transitioning  from  the  2D 
image  slices  to  3D  volumetric  surfaces  requires  that  the  images  be  segmented  to  delineate 
regions  of  interest.  Segmentation  is  an  essential  part  of  visualizing  the  patient  in  3D,  which  is 
typically  done  manually.  Automating  this  time  intensive  method  has  been  attempted  for  specific 
organs  such  as  the  liver  [44,  45],  breast  [46],  bladder,  lungs[47],  or  blood  vessels[48].  Not  all 
targets  are  easily  segmented  through  automation.  The  pancreas,  in  particular,  requires  some 
manual  effort  to  accurately  define  them  [49-51].  When  combining  multiple  modalities,  the 
images  must  also  be  registered  with  each  other  to  form  a  simple  frame  of  reference  [52], 

Taking  these  2D  image  slices  and  presenting  them  in  a  3D  visualization  is  the  first  step 
into  new  methods  of  anatomical  analysis.  Simulations  of  biological  structures  are  of  growing 
interest  for  surgical  planning,  computer-aided  surgery,  and  teaching  aids  [53,  54],  These 
simulations  require  accurate  representation  of  the  patient  physiology  and  incorporation  of  other 
image  sets  to  improve  diagnostic  accuracy  [55],  Here  we  describe  methods  to  develop  a  patient- 
specific  3D  visualization  from  CT  image  slices  overlaid  with  18FDG  -PET,  and  independent 
evaluations  of  the  usefulness  of  the  presentation  by  a  group  of  six  surgeons. 

Methods 

Protecting  patient  health  information 

Anonymous  use  of  CT  and  PET  data  from  deceased  Patient  A,  and  questionnaires 
evaluating  usefulness,  were  approved  by  the  TJU IRB  (09E.407)  and  the  USAMRMC  HRPO  (A- 
15712.2).  Files  with  patient  data  were  anonymized  before  data  manipulation  began. 

3D  Visualizations 

Visualizations  of  the  Patient  A  thorax  were  carried  out  with  Amira  5  (Visage  Imaging, 
San  Diego  CA),  a  3D  visualization,  analysis  and  modeling  program  that  takes  a  modular  and 
object-oriented  approach  to  data  visualization  and  analysis.  The  visualizations  were  based  upon  a 
total  of  three  sets  of  data  for  Patient  A:  one  non-contrast  CT  at  1  mm  resolution,  one  PET  image 
at  5  mm  resolution  co-registered  with  the  non-contrast  CT,  and  one  intravenous  contrast  CT  at  1 
mm  resolution. 

A  series  of  steps  must  be  taken  to  transfer  the  image  information  in  the  2D  CT  image 
slices  into  3D  volumes.  Figure  1  presents  a  flow  diagram  of  the  steps  required  for  translating  the 
initial  2D  images  into  the  3D  visualization. 
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Two  versions  of  the  visualization  were  developed  for  subsequent  evaluation  by  surgeons. 
The  first  version  (1.2)  was  based  solely  upon  the  PET/CT  images  and  had  no  surface  refinement. 
The  segmentation  of  this  data  set  did  not  include  a  complete  vascular  network,  but  required  no 
registration  of  the  PET  image,  since  they  were  already  pre-aligned.  After  evaluation  of  the  first 
version,  the  second  version  (1.3)  was  developed  using  the  contrast  CT  images,  which  included  a 
more  complete  vascular  network.  This  version  required  that  the  PET  image  be  aligned  with  the 
contrast  CT  image. 

Image  segmentation 

Segmentation  of  the  stack  of  2D  CT  image  slices  was  accomplished  semi-automatically 
using  a  mixture  of  the  Amira  5  'magic  wand'  tool,  the  'blow  tool',  and  manual  segmentation. 
Segmentation  of  each  image  slice  in  the  stack  yields  a  matrix  of  points,  or  label  field,  that 
defines  which  pixels  are  associated  with  each  other.  The  'magic  wand'  tool  uses  a  region  growing 
function.  When  the  user  selects  a  voxel,  an  area  containing  that  voxel  and  any  number  of  other 
voxels  whose  intensities  lie  within  the  users  defined  ranges  are  selected.  Lines  can  be  drawn  to 
limit  the  extent  of  the  growth.  Similarly,  the  'blow  tool'  was  another  region  growing  method  that 
increases  as  the  mouse  moves  away  from  the  initially  clicked  point.  The  region  grows  in  area  of 
similar  intensities,  stopping  where  the  values  change  abruptly,  i.e.,  edges. 

The  segmented  images  were  then  refined  using  the  ‘smooth  label’  and  ‘remove  islands’ 
options.  The  ‘smooth  label’  tool  uses  a  modified  Gaussian  filter  to  smooth  the  regional 
boundaries,  removing  any  cusps  that  appear  in  the  surface.  The  ‘remove  islands’  tool  finds 
isolated  regions  not  connected  to  the  larger  region  and  removes  them  from  the  segmented  label. 
The  ‘remove  islands’  tool  can  also  find  holes  within  the  region  and  add  them  to  the  label. 

Segmenting  of  the  PET  images  was  done  with  the  'magic  wand'  tool.  The  high  contrast 
between  the  18FDG  uptake  region  of  the  tumor  vs.  the  surrounding  normal  tissue  it  made  it 
possible  to  choose  a  threshold  that  would  pick  only  regions  of  high  intensity. 

Surface  reconstruction 

Each  stack  of  fully  segmented  Patient  A  thoracic  image  slices  were  reconstructed  into  a 
3D  surface  with  an  Amira  5  SurfaceGen  module  attached  to  each  2D  label  field.  The  SurfaceGen 
module  computes  a  triangular  approximation  of  the  surface  from  the  2D  label  fields  and  the 
interfaces  between  differing  regions.  These  base  3D  visuals  have  a  large  number  of  triangles  and 
can  appear  rather  rough.  One  example  would  be  the  liver  consisting  of  129,325  points  and 
258,622  triangles.  The  entire  rendering  of  the  patient  had  1,216,386  triangles  over  12  models: 
skeleton,  liver,  intestines,  duodenum,  right  kidney,  left  kidney,  stomach,  spleen,  pancreas, 
adrenal  glands,  veins,  and  arteries.  As  an  initial  step  in  smoothing  the  visuals,  the  'smoothing' 
option  of  the  SurfaceGen  module  was  set  to  unconstrained  smoothing  and  the  SmoothKemelSize 
variable  was  set  to  3. 

Surface  quality  and  refinement 

The  Amira  5  Simplification  editor  was  applied  to  the  thoracic  base  3D  visuals  to  improve 
the  quality  and  reduce  the  number  of  triangles  forming  the  visuals.  The  Simplification  editor 
uses  an  edge  collapsing  algorithm  to  reduce  edges  of  the  surface  to  points,  while  preserving  the 
original  shape  of  the  surface  by  minimizing  the  error  criterion.  The  3D  surfaces  were  then  re¬ 
meshed  to  improve  their  appearance  using  the  RemeshSurface  module.  For  remeshing,  an 
isotropic  vertex  placement  and  a  50%  reduction  in  the  number  of  triangles  was  used  to  achieve  a 
higher  triangle  quality  and  modest  reduction  in  the  number  of  triangles.  At  this  point  the  liver 
was  simplified  to  9002  points  and  18000  triangles  before  being  remeshed  to  4501  points  and 
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8998  triangles.  In  version  1.3,  the  entire  rendering  contained  225,221  triangles  with  the  same 
number  of  organs  being  represented. 

Registration 

Registration  of  multiple  images  is  necessary  for  studies  of  a  subject  over  time,  when 
comparing  different  modalities,  matching  an  image  volume  with  its  model,  and  aligning  a 
template  with  the  patient’s  image.  Aligning  multiple  images  correctly  is  a  difficult  task  due  to 
organ  motion  during  respiration,  patient  re-positioning,  organ  changes  over  time  due  to  disease, 
deformation  of  target  organs,  and  many  more.  Automated  registration  exists  for  certain 
applications,  such  as  the  lung  or  arteries,  but  is  not  amenable  for  use  with  the  pancreas,  due  to 
similar  intensities  in  the  voxels  between  the  pancreas  and  neighboring  organs. 

A  hybrid  of  automated  and  interactive  registration  was  used  with  the  PET/CT,  CT-only, 
and  contrast  CT  data  sets.  The  focus  of  the  registration  was  to  align  the  PET  images  of  the 
pancreas  so  that  they  could  be  overlaid  with  the  contrast  CT  model.  The  CT  without  contrast, 
which  was  initially  aligned  with  the  PET  images,  was  aligned  to  the  contrast  CT  before  applying 
the  transformation  to  the  PET  images.  Both  the  CT  data  and  the  3D  renderings  were  used  for  the 
registration. 

Initially,  automated  registration  with  the  AffineRegistration  module  aligned  the  images 
roughly.  The  AffineRegistration  module  uses  the  mean  squared  differences  between  gray  values 
of  the  model  and  reference  as  the  metric  for  alignment  and  scaling,  prior  to  further  refinement. 
Interactive  registration  was  then  carried  out  to  refine  the  alignment  and  scale  of  the  images. 
Amira  5  provides  9-parameters  with  which  to  adjust  the  image  transformation;  translation, 
rotation  axis,  and  scale  factor. 

The  final  stage  of  registration  of  the  3D  surfaces  of  the  pancreas  utilized  the 
AlignSurface  module.  This  module  has  three  strategies  for  aligning  the  surface,  using  the  surface 
points,  the  center  of  mass,  or  the  principal  axes  of  the  inertia  tensor.  Three  types  of 
transformation  can  be  specified:  rigid  alignment,  rigid  alignment  with  uniform  scaling,  or  a 
flexible  affine  transformation.  We  used  the  surface  points  with  a  rigid  alignment  to  align  the 
pancreas  of  the  non-contrast  CT  with  that  of  the  contrast  CT. 

Once  the  two  renderings  were  aligned  with  each  other,  the  transformation  could  be 
applied  to  the  PET  images.  Since  the  PET  images  were  initially  aligned  with  those  of  the  non¬ 
contrast  CT,  the  transformations  applied  to  the  CT  images  were  repeated  with  the  PET  images. 

Evaluations 

Six  surgeons  from  Thomas  Jefferson  University  Hospital  volunteered  to  evaluate  both 
versions  of  the  Patient  A  thoracic  visualization.  These  surgeons  varied  from  general  surgery  to 
thoracic  surgery.  The  surgeons  were  shown  how  to  manipulate  the  image,  especially  how  to  strip 
off  overlaying  tissues  and  organs  overlaying  the  pancreas. 

The  evaluation  consisted  of  multiple  questions  using  a  scale  from  1  to  5:  1  =  strongly 
disagree,  2  =  disagree,  3  =  neither  agree/disagree,  4  =  agree,  and  5  =  strongly  agree.  Blank  space 
was  also  provided  for  each  surgeon  to  leave  comments  on  the  visual  appearance  and  usefulness 
of  the  presentation,  plus  any  overall  comments. 

The  evaluation  and  questionnaire  were  done  in  the  presence  of  the  interviewer  to  help 
answer  any  questions  about  the  visuals,  manipulating  Amira  5,  or  the  questionnaire.  This  mode 
facilitated  the  surgeons’  usage  of  Amira  5,  independent  of  any  prior  knowledge  of  the  program, 
and  yielded  a  more  reliable  evaluation  of  the  system. 


Results 
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Evaluation  was  carried  out  to  gather  information  on  what  aspects  and  features  of  the 
thoracic  visualizations  the  surgeons  found  useful.  Snapshots  from  version  1 .2  are  shown  in 
Figures  2  and  3,  while  snapshots  from  version  1.3  appear  in  Figures  4  and  5.  There  was  an 
overall  improvement  in  the  ratings  of  the  visualizations  going  from  version  1.2  to  1.3. 

An  example  of  the  FDG  -PET  overlay  is  given  in  Figure  6.  Because  of  the  lower 
resolution  in  the  PET  images  (5  mm),  parts  of  the  surface  representation  of  the  tumor  formed 
from  the  PET  data  do  not  exactly  replicate  the  contours  of  the  pancreas. 

Figures  7  presents  the  average  scores  from  the  surgeon  evaluations  for  each  of  the  15 
questions  asked  for  versions  1.2  and  1.3.  Averaging  surgeon  responses  to  the  15  questions, 
version  1.2  scored  3.6  ±  0.5,  while  version  1.3  received  a  score  of  4.4  ±  0.4.  The  overall  averages 
of  the  questionnaire  illustrate  greater  appreciation  of  version  1.3,  while  the  standard  deviations 
reveal  greater  consensus  in  evaluating  version  1.3.  A  paired-samples  t-test  was  done  to  compare 
the  average  of  the  scores  given  to  each  question  in  version  1.2  and  version  1.3.  There  was  a 
statistically  significant  difference  in  these  scores  for  version  1.2  (M  =  3.6,  SD  0.5)  and  version 
1.3  (M  =  4.4,  SD  =  0.4);  t(14)  =  -7.056,  p  <  0.001.  Our  results  show  that  the  improvements  of 
version  1 .3  had  a  positive  impact  on  the  scoring  of  the  visualizations. 

In  both  versions,  the  surgeons  found  the  hardware  and  image  manipulation/re-positioning 
easy  to  use.  Hardware  was  assigned  an  average  score  of  4.5  ±  0.6  for  both  versions.  Image 
manipulation  scores  increased  from  4.2  ±  1.0  for  version  1.2  to  4.5  ±  0.6  for  version  1.3. 

The  accuracy  of  the  segmentations,  their  rendering,  and  the  surgeons’  satisfaction  with 
the  level  of  detail  all  improved  going  from  version  1.2  to  1.3.  For  version  1.2,  4  out  of  6 
surgeons  agreed  with  the  accuracy  of  the  segmented  images,  with  one  neither  agreeing  nor 
disagreeing,  and  one  strongly  disagreeing,  for  an  average  score  of  3.7  ±  1.5.  The  colors  and 
textures  used  were  quite  similar  with  4  of  6  agreeing,  one  neither  agreeing  nor  disagreeing,  and 
one  disagreeing,  for  an  average  score  of  3.8  ±  1.2.  The  average  score  for  the  level  of  detail  was 
3.2  ±1.5  with  only  half  the  surgeons  being  satisfied  with  it,  one  neither  agreeing  nor  disagreeing, 
one  disagreeing,  and  the  last  strongly  disagreeing. 

Version  1.3,  on  the  other  hand,  received  scores  of  4.5  ±  0.8,  4.7  ±  0.8  and  4.0  ±  1.1  for 
the  segmentation,  rendering,  and  level  of  detail.  Only  one  surgeon  couldn't  agree  or  disagree 
with  the  segmentations  and  rendering,  but  disagreed  with  the  level  of  detail. 

Overall  the  surgeons  assigned  version  1.2  a  score  of  3.5  ±1.2  for  the  model  providing 
adequate  reference  to  surrounding  structures,  and  a  score  of  3.5  ±1.2  For  how  well  it  matches 
expectations  in  the  OR  while  version  1.3  faired  much  better  with  a  score  of  4.7  ±  0.5  and  4.0  ± 

1 . 1  respectively. 

While  viewing  only  the  3D  CT  renderings  of  the  Patient  A  thorax,  the  surgeons  were 
asked  to  rate  their  understanding  of  the  tumor  with  its  surrounding  organs  and  if  this  view  would 
change  how  they  might  plan  to  approach  this  tumor.  Version  1.2  scored  an  average  of  3.8  ±1.2 
on  understanding  of  the  tumor  and  its  surroundings,  but  a  2.7  ±  1 .0  on  changing  how  they  might 
plan  to  approach  it.  For  version  1.3,  however,  average  scores  of  4.3  ±  0.8  and  3.8  ±  0.8  were 
given  for  understanding  of  the  tumor  with  its  surrounding  organs,  and  if  this  view  would  change 
how  they  might  plan  to  approach  this  tumor,  respectively. 

Viewing  the  3D  CT  renderings  with  the  PET  overlay  was  rated  similarly  to  those  without 
the  overlay  for  understanding  of  the  tumor  with  its  surrounding  organs,  and  if  this  view  would 
change  how  they  might  plan  to  approach  this  tumor.  Version  1 .2  received  scores  of  3.3  ±  1.0  and 
2.7  ±  1.2  for  understanding  and  approach,  while  version  1.3  was  scored  at  4.2  ±  0.8  and  3.6  ± 

0.9. 
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Four  questions  were  asked  on  the  usefulness  of  the  visualizations.  The  first  question 
asked  if  the  surgeon  would  want  to  use  this  3D  image  to  plan  an  operation  for  a  patient  with  this 
specific  tumor.  The  surgeons  scored  version  1.2  at  3.7  ±  1.8,  and  version  1.3  at  4.8  ±  0.4.  In  the 
second  question,  the  surgeons  were  asked  if  they  would  want  to  use  these  images  with  the  PET 
overlay  to  plan  an  operation  for  a  patient  with  this  specific  tumor.  They  scored  version  1.2  at  3.7 
±  1.8,  and  version  1.3  at  4.3  ±  1.2.  Next,  the  surgeons  were  asked  if  they  would  like  a  system 
such  as  this  available  in  the  OR  for  reference  during  an  actual  operation.  Few  agreed  with 
version  1.2,  giving  it  a  score  of  3.5  ±  1.6,  but  for  version  1.3  most  strongly  agreed,  yielding  an 
average  of  4.7  ±  0.5.  The  final  question  asked  if  this  system  would  help  residents/assistants 
better  prepare  for  the  operation.  In  both  versions  the  surgeons  agreed,  giving  version  1.2  a  score 
of  4.0  ±1.6  and  version  1.3  a  score  of  4.8  ±  0.4. 

Each  surgeon  was  given  space  on  each  questionnaire  to  leave  comments  on  how  the 
visual  models  could  be  improved  and  what  would  make  the  models  more  useful.  In  version  1 .2 
every  surgeon  who  commented  asked  for  a  more  detailed  inclusion  of  the  vascular  system.  A  few 
left  suggestions  for  more  textures  and  detail  in  the  models.  With  version  1.3,  an  appreciation  for 
the  inclusion  of  the  vascular  system  was  seen  in  many  of  the  comments.  Some  asked  for 
improvements  in  the  fine  details  of  the  tumor  and  blood  vessels.  In  this  version  the  blood  vessels 
were  given  slightly  different  hues  to  help  differentiate  the  branches.  One  surgeon  proposed 
making  all  the  vessels  one  color,  while  another  wanted  to  be  able  to  remove  specific  vessels  as 
desired.  One  other  comment  mentioned  that  determining  tumor  invasion  of  adjacent  structures 
was  difficult  to  determine. 

Discussion 

The  evaluations  of  the  two  visualization  versions  were  positive  overall,  but  the 
improvements  in  version  1.3  yielded  much  better  scores  from  the  surgeons.  Changes  from 
version  1.2  to  1.3  included  a  more  detailed  vascular  system  using  the  contrast  enhanced  CT 
images  and  reduction  of  the  triangle  counts  in  the  models  to  accelerate  computer  performance. 
Color-coded  distributions  of  the  scores  assigned  by  the  surgeons  to  each  question  for  version  1.2 
and  1.3  are  shown  in  Figures  8  and  9. 

In  version  1.2,  a  broad  spread  of  opinions  was  displayed  in  the  surgeon  responses.  Over 
all  the  questions,  60%  of  the  surgeons  agreed  or  strongly  agreed  while  26%  disagreed  or  strongly 
disagreed.  14%  of  the  questions  received  neither  agreement  nor  disagreement. 

Of  all  the  questions,  only  the  ease  and  comfort  of  use  for  the  display  hardware  received 
unanimous  agreement  with  three  agreeing  and  three  agreeing  strongly.  Most  found  the  images 
easy  to  manipulate  or  reposition,  scored  at  4.2  ±  1 .0,  with  three  strongly  agreeing,  one  agreeing, 
and  two  having  neither  agreeing  or  disagreeing.  Four  surgeons  felt  strongly  that  this  version 
would  be  helpful  for  residents  and  assistants  to  better  prepare  for  the  operation  with  two 
disagreeing,  with  an  overall  score  of  4.0  ±  1.6. 

The  two  questions  on  viewing  the  3D  models  with  and  without  the  PET  data  generated 
broad  disagreement,  with  an  average  score  of  2.7  ±1.21  and  2.7  ±  1.03  respectively.  Looking 
over  the  comments,  a  clear  desire  from  the  surgeons  was  to  have  more  detail  included  in  the 
models.  In  particular  they  wanted  so  see  the  vascular  structures  around  the  tumor. 

The  majority  of  scores  for  version  1.2  ranged  from  3.8  to  3.2,  with  usually  four  of  the  six 
surgeons  agreeing  with  the  question.  The  highest  scored  questions  in  version  1.2  concerned  the 
color  and  textures  used  for  the  models  as  well  as  whether  viewing  the  3D  model  produced  better 
understanding  of  the  tumor  and  its  surroundings.  On  the  other  hand,  getting  a  better 
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understanding  of  the  tumor  and  its  surroundings  by  overlapping  the  3D  models  with  the  PET 
image  received  only  a  score  of  3.3  ±  l.O.The  surgeons  were  also  not  satisfied  with  the  level  of 
detail  that  was  presented  in  version  1.2,  giving  it  a  score  of  3.2  ±  1.5. 

Scores  for  version  1.3  showed  a  vast  improvement  in  the  surgeons’  opinions  of  the 
visualization.  Over  all  the  questions,  the  surgeons  agreed  or  strongly  agreed  with  85%,  neither 
agreed  nor  disagreed  with  10%,  disagreed  with  3%,  and  left  no  comment  with  2%. 

Many  of  the  questions  received  unanimous  agreement  from  the  surgeons.  Two  received  a 
high  agreement  score  of  4.8.  Five  out  of  six  surgeons  strongly  agreed  that  they  would  want  to  us 
this  3D  image  to  plan  an  operation  for  a  patient  with  this  specific  tumor.  Similarly,  they  felt 
strongly  that  this  system  would  help  residents  and  assistants  better  prepare  for  the  operation. 

Only  two  questions  in  version  1.3  received  an  average  score  below  4.0.  Whether  viewing 
the  3D  CT  models  alone  changed  how  they  would  plan  to  approach  the  tumor  received  a  score  of 
3.8  ±  0.8,  while  including  the  PET  data  received  a  score  of  3.6  ±  0.9.  This  is  an  improvement 
over  the  scores  from  version  1.2,  demonstrating  that  inclusion  of  the  vascular  systems  is 
important  for  planning,  but  also  indicating  that  further  improvements  are  needed  for  the  3D 
models  to  be  most  useful.  For  the  whole  of  version  1.3,  only  three  times  did  a  question  receive  a 
disagreeing  score  and  none  strongly  disagreed.  These  three  included  being  satisfied  with  the 
level  of  detail,  the  overall  3D  image  appearing  realistic,  and  if  the  surgeon  would  want  to  use 
this  system  with  PET  overlay  to  plan  an  operation  for  a  patient  with  this  specific  tumor.  This  is  a 
marked  improvement  over  the  total  of  22  disagrees  and  strongly  disagrees  from  version  1 .2. 

The  rest  of  the  questions  in  version  1.3  received  an  average  score  ranging  from  4.7  to  4.0. 
Three  received  scores  of  4.7,  including  the  color  and  textures  of  the  models,  models  providing 
adequate  reference  to  surrounding  structures,  and  the  surgeons  desire  to  have  this  system 
available  in  the  OR  for  reference  during  an  actual  operation. 

Three  questions  displayed  large  improvement,  increasing  by  1 .2,  with  the  upgrades  in  the 
model  from  version  1.2  to  1.3.  Inclusion  of  the  vascular  system  provided  the  needed  references 
to  structures  surrounding  the  tumor.  That  upgrade  also  markedly  improved  how  just  by  viewing 
the  3D  model  their  plan  of  approaching  this  tumor  would  change.  The  improvements  also  made 
this  model  attractive  for  planning  an  operation  on  a  patient  with  this  specific  tumor. 

The  changes  made  between  the  two  versions  raised  the  surgeons’  opinions  of  the  models, 
but  further  improvements  are  needed.  In  particular,  even  with  the  vascular  system  included  in  the 
visualization,  the  surgeons  saw  only  modest  benefit  for  planning  how  to  approach  the  pancreatic 
tumor.  One  idea,  suggested  by  a  surgeon,  would  be  to  include  visual  warnings  for  adjacent 
structures  that  are  close  to  the  tumor.  This  could  draw  the  surgeon’s  attention  to  structures,  such 
as  blood  vessels,  near  the  tumor  that  could  be  of  concern  for  unintended  cutting.  Another 
suggestion  was  to  have  the  color  of  the  vascular  system  be  dependent  on  the  theoretical 
oxygenation  level  in  the  blood. 

Conclusions 

This  work  describes  a  method  for  rendering  individual  patient  data  as  3D  surfaces  and  for 
surgeon  evaluation  of  the  usefulness  of  the  resulting  visualization.  Regular  CT  and  contrast  CT 
images  were  both  used  to  generate  the  surface  meshes.  Inclusion  of  PET  images  clearly  defined 
what  regions  of  the  model  contained  aggressive  tumor  cells.  The  evaluating  surgeons  found  these 
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visualizations  to  be  useful  tools  for  planning  an  operation,  as  a  reference  in  the  operating  room 
during  surgery,  and  as  a  reference  for  residents  and  assistants. 

The  next  step  in  the  development  of  this  visualization  system  is  to  transfer  the  3D  models 
to  the  Simulation  Open  Framework  Architecture  (SOFA)[56],  By  integrating  a  physical  response 
model  with  the  visualization,  the  surgeons  will  be  able  to  interact  directly  with  the  models.  A 
haptic  interface  will  also  be  introduced  to  provide  tactile  feedback  to  the  surgeons  so  that  they 
can  practice  palpation  of  the  tumor  region  before  ever  having  to  make  an  incision.  A  touch-and- 
feel  simulation  of  a  specific  patient  could  be  a  useful  practice  tool  for  surgeons  prior  to  making 
the  first  incision,  and  could  be  a  useful  tool  for  teaching  medical  students  and  residents  important 
skills. 
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Figure  1:  Flowchart  depicting  the  process  for  the  p. 
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Figure  2:  Volumetric  rendering  of  Patient  A  in  version  1.2.  Anatomy  displayed  in  this  rendering 
are  a)  skeleton,  b)  liver,  c)  intestines,  d)  stomach,  e)  pancreas,  and  f)  aorta. 
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Figure  3:  Visualization  1.2  with  some  organs  stripped  away  to  more  fully  display  the  pancreatic 
tumor  (black  arrow).  Organs  displayed  in  this  rendering  are  a)  skeleton,  b)  kidneys,  c) 
duodenum,  d)  spleen,  e)  adrenal  gland,  f)  aorta,  g)  vena  cava,  h)  superior  mesenteric  artery,  and 
i)  pancreas. 
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Figure  4:  Volumetric  rendering  of  Patient  A  in  version  1.3.  Anatomy  displayed  in  this  rendering 
are  a)  skeleton,  b)  liver,  c)  intestines,  d)  stomach,  e)  right  gastroepiploic  vein,  and  f)  common 
iliac  artery. 
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Figure  5:  Visualization  1.3  with  some  organs  stripped  away  to  more  fully  display  the  pancreatic 
tumor  (black  arrow).  Organs  displayed  in  this  rendering  are  a)  skeleton,  b)  kidneys,  c) 
duodenum,  d)  spleen,  e)  aorta,  f)  vena  cava,  g),  pancreas  h)  superior  mesenteric  artery,  i) 
superior  mesenteric  vein,  j)  right  gastroepiploic  vein,  k)  portal  vein,  and  other  blood  vessels. 
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Figure  6:  Image  depicting  a  transparent  overlay,  in  version  1.2,  of  the  a)  pancreas  over  the 
surface  rendering  of  high  uptake  region  of  18FDG  in  the  b)  pancreatic  tumor  (orange). 
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Average  Score 

a)  0.0  1.0  2.0  3.0  4.0  5.0 


1.  The  display  hardware  is 
easy/comfortable  to  use 


2.  I  found  it  easy  to 
manipulate/re-position  the  image 


3.  The  organs/structures  are 
accurately  represented 
(accuracy  of  segmentation), 

4.  Colors/textures  are 
appropriate  (accuracy  of 
rendering) 


5.  I  am  satisfied  w  ith  the  level 
of  detail  that  is  presented. 


6.  The  model  provides  me  w  ith 
adequate  reference  to 
surrounding  structures 

7.  The  overall  3D  image 
appears  realistic  (matches 
w  hat  I  expect  to  see  in  OR) 


□  Version  1 .2  □  Version  1 .3 
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Average  Score 
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8.  By  simply  view  ing  the 
image(s)  in  the  3D  model,  I  get  a 
better  understanding  of  the 
tumor  and  its  relationship  to  the 
surrounding  organs. 

9.  By  simply  view  ing  the 
image(s)  in  the  3D  model,  my 
plan  for  how  to  approach  this 
tumor  changed,  (as  compared 
to  traditional  CT  images) 

10.  By  overlaying  the  PET  data 
in  the  3D  model,  I  get  a  better 

understanding  of  the  tumor  and 
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surrounding  organs. 

1 1 .  By  overlaying  the  PET  data 
in  the  3D  model,  my  plan  for 
how  to  approach  this  tumor 
changed,  (as  compared  to 

traditional  CT  images) 


□  Version  1.2  aversion  1.3 
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Average  Score 
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12.  I  would  want  to  use  this  3D 
image  to  plan  an  operation  for  a 
patient  w  ith  a  specific  tumor 


13.  I  w  ould  w  ant  to  use  this  3D 
image  w  ith  PET  overlay  to  plan 
an  operation  for  a  patient  w  ith  a 
specific  tumor 


14.  I  would  like  to  have  this 
system  available  to  me  in  the 
OR,  for  the  reference  during  an 
actual  operation 


15.  I  believe  that  this  system 
would  help  residents/assistant 
better  prepare  for  the 
operation. 


□  Version  1.2  aversion  1.3 


Figure  7:  Average  scores  for  version  1.2  and  1.3  of  the  visualization,  with  error  bars,  for 
questions  on  a)  the  ease  of  use  and  accuracy  of  models,  b)  whether  viewing  these  models,  with 
or  without  PET  changed  their  understanding  of  the  tumor,  and  c)  what  this  type  system  would  be 
applicable  to. 
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1 .  The  display  hardware  is  easy/comfortable  to  use 

2.  I  found  it  easy  to  manipulate/re-position  the  image 

3.  The  organs/structures  are  accurately  represented  (accuracy  of  segmentation), 

4.  Colors/textures  are  appropriate  (accuracy  of  rendering) 
5. 1  am  satisfied  with  the  level  of  detail  that  is  presented. 

6.  The  model  provides  me  with  adequate  reference  to  surrounding  structures 

7.  The  overall  3D  image  appears  realistic  (matches  what  I  expect  to  see  in  OR) 

8.  By  simply  viewing  the  image(s)  in  the  3D  model,  I  get  a  better  understanding  of  the  tumor 
and  its  relationship  to  the  surrounding  organs. 

9.  By  simply  viewing  the  image(s)  in  the  3D  model,  my  plan  for  how  to  approach  this  tumor 
changed,  (as  compared  to  traditional  CT  images) 

10.  By  overlaying  the  PET  data  in  the  3D  model,  I  get  a  better  understanding  of  the  tumor 
and  its  relationship  to  the  surrounding  organs. 

1 1 .  By  overlaying  the  PET  data  in  the  3D  model,  my  plan  for  how  to  approach  this  tumor 
changed,  (as  compared  to  traditional  CT  images) 

12.  I  would  want  to  use  this  3D  image  to  plan  an  operation  for  a  patient  with  a  specific  tumor 

13.  I  would  want  to  use  this  3D  image  with  PET  overlay  to  plan  an  operation  for  a  patient 

with  a  specific  tumor 

14.  I  would  like  to  have  this  system  available  to  me  in  the  OR,  for  the  reference  during  an 

actual  operation 

15. 1  believe  that  this  system  would  help  residents/assistant  better  prepare  for  the  operation. 


□  5  (strongly  agree)  D4  (agree)  D3  (neither  agree  nor  disagree)  M2  (disagree)  HI  (strongly  disagree) 


Figure  8:  Distribution  of  surgeon  scores  in  version  1.2  for  each  question  and  the  percentage  of 
surgeons  that  assigned  that  score. 
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1 .  The  display  hardware  is  easy/comfortable  to  use 

2.  I  found  it  easy  to  manipulate/re-position  the  image 

3.  The  organs/structures  are  accurately  represented  (accuracy  of  segmentation), 

4.  Colors/textures  are  appropriate  (accuracy  of  rendering) 
5. 1  am  satisfied  with  the  level  of  detail  that  is  presented. 

6.  The  model  provides  me  with  adequate  reference  to  surrounding  structures 

7.  The  overall  3D  image  appears  realistic  (matches  what  I  expect  to  see  in  OR) 

8.  By  simply  viewing  the  image(s)  in  the  3D  model,  I  get  a  better  understanding  of  the 
tumor  and  its  relationship  to  the  surrounding  organs. 

9.  By  simply  viewing  the  image(s)  in  the  3D  model,  my  plan  for  how  to  approach  this 
tumor  changed,  (as  compared  to  traditional  CT  images) 

10.  By  overlaying  the  PET  data  in  the  3D  model,  I  get  a  better  understanding  of  the  tumor 
and  its  relationship  to  the  surrounding  organs. 

1 1.  By  overlaying  the  PET  data  in  the  3D  model,  my  plan  for  how  to  approach  this  tumor 
changed,  (as  compared  to  traditional  CT  images) 

12.  I  would  want  to  use  this  3D  image  to  plan  an  operation  for  a  patient  with  a  specific 
tumor 

13.  I  would  want  to  use  this  3D  image  with  PET  overlay  to  plan  an  operation  for  a  patient 
with  a  specific  tumor 

14. 1  would  like  to  have  this  system  available  to  me  in  the  OR,  for  the  reference  during  an 
actual  operation 

15.  I  believe  that  this  system  would  help  residents/assistant  better  prepare  for  the 
operation. 


□  5  (strongly  agree)  D4  (agree)  D3  (neither  agree  nor  disagree)  112  (disagree)  11  (strongly  disagree)  DNA 


Figure  9:  Distribution  of  surgeon  scores  in  version  1.3  for  each  question  and  the  percentage  of 
surgeons  that  assigned  that  score. 


